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Topics 

Introduction  (10  min) 

Overview  of  the  Steps  to  Build  PPMs  (80  mins) 

-  Preparing  to  Build  PPMs 

-  Developing  PPMs 

-  Using  PPMs 

Exercise  1 :  Constructing  a  Product  Business  Case  with  Monte  Carlo  Simulation 
and  Optimization  (40  mins) 

Exercise  2:  Scheduling  Projects  with  Monte  Carlo  Simulation  and  Optimization 

(30  mins) 

Exercise  3:  Predicting  Product  Requirements  Change  with  Linear  Regression 

(30  mins) 

Exercise  4:  Predicting  Delivered  Defects  with  Dummy  Variable  Regression 

(30  mins) 

Exercise  5:  Predicting  Customer  Satisfaction  using  Ordinal  Logistic  Regression 
Questions  (30  mins) 
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What  is  a  PPM? 


OPPSP  1.5 

•  PPMs  are  used  to  estimate  or  predict  the  value  of  a  process- 
performance  measure  from  the  values  of  other  process,  product, 
and  service  measurements 

•  PPMs  typically  use  process  and  product  measurements  collected 
throughout  the  life  of  the  project  to  estimate  progress  toward 
achieving  objectives  that  cannot  be  measured  until  later  in  the 
project’s  life 

Glossary 

•  A  description  of  the  relationships  among  attributes  of  a  process  and 
its  work  products  that  is  developed  from  historical  process- 
performance  data  and  calibrated  using  collected  process  and 
product  measures  from  the  project  and  that  is  used  to  predict  results 
to  be  achieved  by  following  a  process 
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Purpose  and  Usage  of  Process  Performance 
Models  at  the  Organizational  Level 


•  Identifying  Organizational 
Priorities  for  Quality  and 
Process  Performance 

•  Establishing  and 
Revising  Organizational 
Quality  and  Process 
Performance  Objections 


•  Identifying  Process 
Performance  Measures 


•  Defining  New  Process 
Performance  Baselines 


•  Analyzing  Process  and 
Technology  Improvement 
Proposals 

•  Identifying  Process  and 
Technology  Improvement 
Proposals 

•  Prioritizing  Candidate 
Process  and  Technology 
Improvements  for 
Deployment 
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Purpose  and  Usage  of  Process  Performance 
Models  at  the  Project  Level 


Software 

Design 


Requirementsi 
Elicitation 


Project 

Planning 


Software  Unit  Testing 


Systems 

Testing 


Customer 

Acceptance 

Testing 


Project 

Start 

Proposal 


Project 

Finish 
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Healthy  Ingredients  of  CMMI  Process 
Performance  Models 

Statistical,  probabilistic  or  simulation  in  nature 
Predict  interim  and/or  final  project  outcomes 

Use  controllable  factors  tied  to  sub-processes  to  conduct  the  prediction 

Model  the  variation  of  factors  and  understand  the  predicted  range  or 
variation  of  the  outcomes 

Enable  “what-if  analysis  for  project  planning,  dynamic  re-planning  and 
problem  resolution  during  project  execution 

Connect  “upstream”  activity  with  “downstream”  activity 

Enable  projects  to  achieve  mid-course  corrections  to  ensure  project 
success 
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All  Models  (Qualitative  and  Quantitative) 


Quantitative  Models  (Deterministic,  Statistical,  Probabilistic) 


Statistical  or  Probabilistic  Models 


Interim  outcomes  predicted 
Controllable  x  factors  involved 


Process  Performance 
Model  - 

With  controllable  x 
factors  tied  to 
Processes  and/or 
Sub-processes 


Only  phases 
or  lifecycles 
are  modeled 


Only 
uncontrollable 
factors  are 
modeled 


Only  final 
outcomes 
are 

modeled 


No 

uncertainty 
or  variation 
modeled 


Anecdotal 
data  and 
biased 
samples 
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Overview  of  the  Steps 
to  Build  PPMs 

-  Preparing  to  Develoc 
PPMs 
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Preparing  to  Develop  PPMs 

1.  Initiating  the  development  of  process  performance  models  from 
a  context  of  the  customer  and  Business  goals 

2.  Using  correct  critical  thinking  and  root  cause  analysis  to  identify 
the  proper  outcomes  and  drivers  of  the  outcomes  (including 
controllable  and  uncontrollable  process  factors) 

3.  Becoming  sensitive  to  the  types  of  issues  and  documentation 
needed  during  the  development  of  the  process  performance 
models 

4.  Addressing  issues  related  to  data  collection,  measurement 
scale,  data  quality  and  integrity,  outliers  and  measurement  error 

5.  Identifying  the  data  types  involved  with  the  outcomes  and 
process  drivers 

6.  Creating  performance  baselines  of  outcomes  and  process 
drivers 

7.  Forming  a  team  to  develop  a  process  performance  model 
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Step  1  -  Identify  or  Reconfirm  Business  Goals 


Opportunities  vs 
Threats 


Goals  to  Defeat 
Barriers  to  Vision 


Opportunity 


Strength  Weakness 


SMART  Business  Goals 

(Specific,  Measurable,  Attainable, 
Relevant,  and  Timely) 
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Step  1  -  Business  Goal  Flowdown  (Y-to-x) 


High  Level  Business  Goals 

(Balanced  Scorecard) 


Subordinate  Business  Goals 

(e.g.,  $  Buckets, 

%  Performance) 


High  Level  Process 

(e.g.,  Organizational  Processes) 


Subordinate 

Processes 

(e.g.,  Down  to  a  Vital  x 
sub-process  to  be 
tackled  by  DMAIC  team) 
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Questions 
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1. 


2. 


3. 


4. 


Are  vour  senior  leaders  defining  business  goals 

rather  than  delegating  goal  definition  to  operational 

levels? 


Do  lower  organizational  levels  redefine  the  higher 

level  goals  in  operational  terms  or  do  they  merely 

block  copy  and  paste  upper  goals? 


Are  you  organization’s  business  goals  SMART? 


Has  vour  organization  ensured  that  process 
performance  baselines  and  models  are  targeted  at 

the  most  important  issues  and  goals? 
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Step  2  -  Identify  the  Sub-Process/Process 
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•  Start  with  the  Organization’s  Business  Objectives 

•  Decompose  to  Quality  and  Process  Performance 
Objectives  (QPPOs) 

•  For  the  QPPOs  that  can  be  Measured  Quantitatively 

•  Perform  Analysis  to  Determine  which  Sub-Process/Process  Drives 
the  Relevant  Objective 

•  Determine  if  Sufficient  Data  is  Available  or  can  be  Obtained  to 
Establish  a  Process  Performance  Baseline(s)  and/or  Build  a 
Process  Performance  Model(s) 
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Step  2  -  Identify  the  Sub-Process/Process 
Example 

•  Given  Organizational  Business  Objectives: 

•  Improve  quality 

•  Improve  cycle  time 

•  Improve  productivity 

•  Translate  to  measureable  QPPOs 

•  Post-delivery  defect  density  of  less  than  0.5  Defects/KSLOC 

•  Achieve  85%  defect  detection  before  System  testing 

•  Ensure  requirements  duration  is  within  15%  of  plan 

•  Achieve  a  5  %  software  productivity  improvement 
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Step  2  -  Examples  of  Outcomes 
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Step  2  -  Identify  Controllable  factors  (x’s)  to 
Predict  Outcome(s)  - 1 
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“Controllable”  implies  that  a  project  has  direct  or  indirect 
influence  over  the  factor  prior  to  or  during  the  project 
execution 


A  common  misconception  is  that  factors  are  not  controllable 
and  thus  disregarded  from  consideration  for  modeling. 
Requires  out-of-the-box  thinking  to  overcome  this.  Some 
organizations  employ  individuals  known  as  “assumption 
busters” 
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SSTC  2010 

Step  2  -  Identify  Controllable  factors  (x’s)  to 
Predict  Outcome(s)  -  2 


As  we  view  process  holistically,  controllable  factors  may  be 
related,  but  not  limited,  to  any  of  the  following: 


•  People  attributes 

•  Environmental  factors 

•  Technology  factors 

•  Tools  (physical  or  software) 

•  Process  factors 

•  Customers 

•  Suppliers 

•  Other  Stakeholders 
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Step  2  -  Examples  of  Controllable  People  x 

faCtOrS  .  _  fnn|C 

Absolute  performance  ot  a  tas 

Trainirig  Variability  of  performance  of  a  task  or  l 


Skills 
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Variability  of  performance  of  a  task  or  topic 
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Experience  Levels 
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Geographic  dispersion  of  Diversity  ot  sta«  ^  and  outiooKs 

Communication  Mechanisms  Various  Team  in 
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rooms 


Step  2  -  Example  of  Controllable  Environmental 
x  Factors  .  _ 
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Step  2  -  Example  of  Controllable  Technology  x 
Factors 
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Step  2  -  Example  of  Controllable  Process  x 

FdCtorS  Quality  of  artifacts 

Resolution  time  of  technical  inquiries  (lnput  to  or  output  from 

Efficiency  of  a  work  task  a  work  task\ 

Compliance  of  a  work  task 


Timeliness  of  Artifacts 


Quality  of  a  work  task 

Timeliness  of  a  work  task 

Measures  of  bureaucracy 

Resource  contention  between  tasks 
Difficulty  of  a  work  task 


Task  Interdependence 
Complexity  of  Artifacts 
.  Readability  of  Artifacts 


Number  of  people  involved  with  a  work  task 


Any  of  the  criteria  for 
good  reqts  statements 
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Peer  Review  Measures  _  . 

Code  measures 

Test  Coverage  Modifications  to  how  work  (static  and  Dynamic) 
Measures  Tasks  are  performed 
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Step  2  -  Example  of  Controllable  Customer, 
Supplier  and  Other  Stakeholder  x  Factors 
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•  Normally  these  are  constraints  placed  by  the  customer  or 
concrete  terms  of  a  contract  or  government  regulation 


•  Can  also  be  factors  for  which  the  project  team  truly  has  no 
direct  nor  indirect  influence  over 


•  Can  be  factors  that  are  unchanging  for  a  given  project  but 
can  be  changed  for  future  projects 


•  Often  includes  external  factors  or  factors  related  to  other 
teams  outside  of  the  project 
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Questions 
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1. 


2. 


3. 


4. 


What  is  a  critical,  high  risk,  uncertain  subprocess 

within  your  organization? 


What  is  a  potential  outcome  performance  measure 

related  to  that  subprocess? 


What  are  2-3  controllable  factors  directly 

influencing  this  outcome  measure? 


Do  you  believe  there  are  any  uncontrollable  factors 

dominating  this  outcome  measure? 
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Step  3  -  Cost  of  Poor  Data  Quality  to  an 
Enterprise  -  Typical  Issues  and  Impacts 


SSTC  2010 


Typical  Issues 

•  Inaccurate  data  [1-5%  of  data  fields  are  erred] 

•  Inconsistencies  across  databases 

•  Unavailable  data  necessary  for  certain  operations  or  decisions 

Typical  Impacts 


Operational 

•  Lowered  customer 
satisfaction 

•  Increased  cost 

•  Lowered  employee 
satisfaction 


Source:  Redman,  1998 


Tactical 

•  Poorer  decision  making  & 
decisions  take  longer 

•  More  difficult  to  implement 
data  warehouses 

•  More  difficult  to  engineer 

•  Increased  organizational 
mistrust 


_ Strategic _ 

•  More  difficult  to  set  strategy 

•  More  difficult  to  execute  strategy 

•  Contribute  to  issues  of  data 
ownership 

•  Compromise  ability  to  align 
organization 

•  Divert  management  attention 
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Step  3  -  Impacts  of  Poor  Data  Quality 


SSTC  2010 


Inability  to 

•  manage  the  quality  and  performance  of  software  or 
application  development 

•  Estimate  and  plan  realistically 

Ineffective 

•  process  change  instead  of  process  improvement 

•  and  inefficient  testing  causing  issues  with  time  to 
market,  field  quality  and  development  costs 

Products  that  are  painful  and  costly  to  use  within  real-life 
usage  profiles 


Bad  Information  leading  to  Bad  Decisions 


Software  Engineering  Institute  CarnegieMellon 


Robert  Stoddard  and  Dave  Zubrow 

©2010  Carnegie  Mellon  University 


30 


SSTC  2010 


Step  3  -  Where  do  Measurement  Errors  come 
Fromi 

Data  Entry  Errors 

-  Manual  data  entry 

-  Lack  of  integrity  checks 

Differing  Operational  Definitions 

-  Project  duration,  defect  severity  or  type,  LOC  definition,  milestone 
completion 

Not  a  priority  for  those  generating  or  collecting  data 

-  Complete  the  effort  time  sheet  at  the  end  of  the  month 

-  Inaccurate  measurement  at  the  source 

Double  Duty 

-  Effort  data  collection  is  for  Accounting  not  Project  Management 

•  Overtime  is  not  tracked 

•  Effort  is  tracked  only  to  highest  level  of  WBS 
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Step  3  -  Where  do  Measurement  Errors  come 
Fronrh 

Dysfunctional  Incentives 

•  Rewards  for  high  productivity  measured  as  LoC/Hr 

•  Dilbert-esque  scenarios 

Failure  to  provide  resources  and  training 

•  Assume  data  collectors  all  understand  goals  and  purpose 

•  Arduous  manual  tasks  instead  of  automation 

Lack  of  priority  or  interest 

•  No  visible  use  or  consequences  associated  with  poor  data  collection  or 
measurement 

•  No  sustained  management  sponsorship 

Missing  data  is  reported  as  a  valid  value 

•  Can’t  distinguish  0  from  missing  when  performing  calculations 
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Indicator  Name/Title 

Objective 

Questions 
Visua 


Perspective 

Input(s) 

Data  Elements 
Definitions 

Data  Collection 
How 

When/How  Ofte 
By  Whom 
Form(s) 

Data  Reporting 

Responsibility 
for  Reporting 

By/To  Whom 

How  Often 


Specify 
Data 
Collection 

Procedures 


Step  3  -  Documenting 
Measurement  Objectives, 
Indicators,  and  Measures 


Data  Storage 
Where 
How 
Security 

Algorithm 


Assumptions  — 
Interpretation  — 

Probing  Questions 
Analysis  — 

Evolution 


J 


pecify 
Analysis 
Procedures 


Analyze 

Data 


Feedback  Guidelines 


X-reference 
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Step  4  -  Identifying  Outliers 


SSTC  2010 


Interquartile  range  description  -  A  quantitative  method  for 
identifying  possible  outliers  in  a  data  set 

Procedure 

•  Determine  1st  and  3rd  quartiles  of  data  set:  Q1 ,  Q3 

•  Calculate  the  difference:  interquartile  range  or  IQR  which  equals 
Q3  minus  Q1 

•  Lower  outlier  boundary  =  Q1  -  1 .5*IQR 

•  Upper  outlier  boundary  =  Q3  +  1 .5*IQR 
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Step  4  -  Interquartile  Range:  Example 


Interquartile  Range 
30-16  =  14 


Procedure 

1 .  Determine  1st  and  3rd 
quartiles  of  data  set:  Q1, 
Q3 

2.  Calculate  the  difference: 
interquartile  range  or  IQR 

3.  Lower  outlier  boundary  = 
Q1  -  1.5*IQR 

4.  Upper  outlier  boundary  = 
Q3  +  1 .5*IQR 


Upper  outlier  boundary 
30  +  1.5*14  =  51 


Lower  outlier  boundary 
16 -1.5*14  = -5 
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Step  4  -  Tips  About  Outliers 
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Outliers  can  be  a  clue  to  process  understanding 
If  outliers  lead  you  to  measurement  system  problems, 

•  repair  the  erroneous  data  if  possible 

•  if  it  cannot  be  repaired,  delete  it 

Charts  that  are  particularly  effective  to  flag  possible  outliers 
include:  box  plots,  distributions,  scatter  plots,  and  control 
charts 

Rescale  charts  when  an  outlier  reduces  visibility  into 
variation. 

Be  wary  of  influence  of  outliers  on  linear  relationships 
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Step  5  -  Types  of  Data 


Nominal 

Attribute 

(aka  categorized  or 
discrete  data) 


Ordinal 

Increasing 

information 

content 


Categorical  data  where  the  order  of  the 
categories  is  arbitrary 


Nominal  data  with  an  ordering;  may 
have  unequal  intervals 


A  B  C 


SIX  SIGMA  ADVANTAGE 

The  Third 


Examples 
Defect  types 
Labor  types 
Languages 

Examples 
Severity  levels 
Survey  choices  1-5 
Experience  categories 


\ 

Continuous 
(aka  variables 
data) 


Interval 

Ratio 


Continuous  data  that  has  equal  intervals; 
may  have  decimal  values 


Interval  data  set 
that  also  has 
a  true  zero  point 


Examples 
Defect  densities 
Labor  rates 
Productivity 
Variance  %’s 
Code  size  SLOC 
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Questions 


SSTC2010 


1.  What  data  type  is  vour  outcome  performance 

measure? 

2.  What  data  type  is  each  of  vour  controllable  and 

uncontrollable  x  factors? 
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Step  6  -  Creating  Process  Performance 
Baselines 

•  Definition:  A  Process  Performance  Baselines  (PPB)  is  a 
documented  characterization  of  the  actual  results  achieved  by 
following  a  process 

•  Therefore  a  PPB  needs  to  reflect  actual  project  performance 

•  CMMI-DEV  OPP  PA  informative  material: 


•  Establish  a  quantitative  understanding  of  the  performance  of  the 
organization’s  set  of  standard  processes  in  support  of  objectives 

•  Select  the  processes  that  summarize  the  actual  performance  of 
processes  in  projects  in  the  organization 

•  Alternatively  Practical  Software  and  Systems  Measurement 
(PSM)  recommends  an  organization  follow  three  basic  steps: 


•  Identify  organization  needs 

•  Select  appropriate  measures 


Integrate  measurement  into  the  process 
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Step  6  -  Creating  Process  Performance 
Baselines  Example 


•  If  we  go  back  to  our  earlier  example  where  we  determined 
that  the  inspection  sub-process  should  be  statistically 
managed 


•  Collect  data  and  Establish  a  PPB  for  the  inspection  sub¬ 
process 
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Step  6  -  Appropriate  Analysis:  Types  of 
Hypothesis  Tests 


SSTC  2010 


^  _  1  r-  Interval  or  Ratio 

,  Data  TVPe  F>  (Parametric  Testsi 

Ordinal 

(Non-Parametric  TestsTi 

Nominal 

Proportion 

. _ A _ _ 

i 

i 

*  Samples 

I'Dafa  groups/ 

’'it3' 

1 

Sample 

2 

Samples 

a+ 

Samples 

Mean  Variance 

Median  Variance  /  Fit 

Similarity  ' 

Similarity 

l-sanple:leEl 

1-sarrple 
Uii-^quare  tes: 

1  sample 
YYilcDxon&fied 

Ranks  tesc 

Kencgorov- 
smirnov 
GdoAksg  uf  Fit 
test 

j 

>2ceTte  / 

Chi- 

Sqjdrt  /' 

/  Binoiual 
/'  SignTes: 

/  =2  cells 

J  ProoDrUons 
test 

/ndependenr 

i-sariple 

Meal  / 

/  Paired 
y  ttest 

/  Paired 

WovnaJ 

Fie:: 

. 

j 

/Levene 

/'  tcSt 

/  Mot  Normal 

.■ 

JridepeHLfcn (  /' 

Mam 

Wh:ney  y 

U  tSt/WilCCKHfl 

nralctied 
/  Paired 

=  Med  rara 

i* 

Siegel-  / 

Tukeytetf /" 

j 

y#  Weses 
y  Jesi 

/  *  Medians 

p 

F  sher  E*acc  test 
(1-way  AN  OVA); 
Chi-  S^te  test 

2  ProoDitions 
test 

ANOVA  [1  &  2  way 
ANOVA;  Balanced 
ANOVA;  GLM1 
MftNOVA  (General 
4  Balanced) 

tferma^ 

,r 

Bartlett,'" 

ICBt 

y  Levene 
test 

y  No?Wthmiaf 

r 

. 

^ndewnden?  / 
Krjstal-Wa  a'- 

1-wa/ 

ANOVA/'  r'edman 
/  2-way 
ANGYA 

Paired 

Van  der 
V/afirden 

Normal  scores  test 

Chi- Snjare  test 

ANOU 

(Analyssof 

Ideas) 
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Step  6  -  Creating  Process  Performance 
Baselines  Misconceptions 

•  We  only  need  one  baseline 

•  Once  we  establish  the  initial  set  of  baselines  we  are  done 

•  One  data  point  constitutes  a  baseline 

•  We  can’t  use  the  baseline  until  it  is  stable 

•  If  the  initial  baseline  is  unstable  we  just  remove  the  data 
points  outside  of  the  control  limits  and  recompute  the  control 
limits  until  we  get  a  plot  that  appears  stable 
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SSTC  2010 

Step  7  -  Skills  Needed  to  Develop  PPMs 

•  Business  Acumen 

•  Product  Expertise 

•  Process  Expertise 

•  Understanding  of  Measurement  and  AnalysisTechniques 

•  Understanding  of  Advanced  Statistical  Techniques 

•  Understanding  of  Quantitative  Management 
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SSTC  2010 


Step  7  -  Forming  the  PPM  Development  Team 

Statistical  Skills 

•  PPM  builder  needs  a  good  understanding  of  statistics  or  Six  Sigma 
Black  Belt  skill  level  or  better 

•  PPM  builder  needs  to  be  an  expert  user  of  the  selected  statistical 
tools 

•  User  of  PPMs  needs  to  be  an  educated  consumer 

Process  knowledge 

•  Build  team  needs  to  understand  the  process 

•  Build  team  needs  to  understand  the  context  in  which  the  PPMs  will 
be  used 
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-  — ;  . 1  Software  Engineering  Institute 


Overview  of  the  Steps 
to  Build  PPMs 

-  Creating  PPMs 
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Creating  PPMs 


SSTC  2010 


1.  Identifying  and  using  the  correct  analytical 
techniques  for  analyzing  baselines,  and  creating 
process  performance  models 

2.  Creating  both  confidence  and  prediction  intervals 
with  the  models 

3.  Validating  and  maintaining  the  process  performance 
models  including  calibration  and  re-confirming  with 
ongoing  process  and  project  data 

4.  Confirming  process  performance  models  meet  the 
established  ingredients  communicated  by  the  SEI, 
either  individually  or  as  a  whole 
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Step  1  -  Select  the  Proper  Analytical  Model 


Types  of  Modeling  Techniques 

*  Statistical  Modeling  and  Regression  Equations 

*  Monte  Carlo  Simulation 

*  Probabilistic  Modeling  including  Bayesian  Belief  Networks 

*  Discrete  Event  Process  Simulation 

*  Other  Advanced  Modeling  Techniques 

•  Markov,  Petri-net,  Neural  Nets,  Systems  Dynamics 
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Step  1  -  Statistical  Regression  Analysis 

Y 


Continuous 


Discrete 


(b 

CD 

AN  OVA 

Chi-Square, 

Vlr 

O 

</> 

and  Dummy 

Logit  & 

■ 

Q 

1 

Variable 

Logistic 

1  = 

Regression 

Regression 

o 

3 

C 

■  HHI 

c 

o 

VO 

Correlation 
&  Linear 
Regression 

Logistic 

Regression 

Software  Engineering  Institute  GarnedeMellon 


Robert  Stoddard  and  Dave  Zubrow 

©2010  Carnegie  Mellon  University 


Step  1  -  Why  Use  Monte  Carlo  Simulation? 
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Use  Monte  Carlo  simulation  to  do  the  following: 

•  Allow  modeling  of  variables  that  are  uncertain  (e.g.,  put  in  a  range  of 
values  instead  of  single  value) 

•  Enable  more  accurate  sensitivity  analysis 

•  Analyze  simultaneous  effects  of  many  different  uncertain  variables 
(e.g.,  more  realistic) 

•  Aid  buy-in  and  acceptance  of  modeling  because  user-provided 
values  for  uncertain  variables  are  included  in  the  analysis 

•  Provide  a  basis  for  confidence  in  a  model  output  (e.g.,  supports  risk 
management) 

•  Increase  the  usefulness  of  the  model  in  predicting  outcomes 
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A 


2  3 


1  2  3  4  5 

Crystal  Ball 
then  allows  the 
user  to  analyze 
and  interpret 
the  final 
distribution  of 
C! 


Crystal  Ball  uses 
a  random  number 
generator  to 
select  values  for 
A  and  B 


B  = 


c 


123456789  10 
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2  3 


1  I  2  3 


1  2  3  4  5 


Crystal  Ball 
causes  Excel  to 
recalculate  all 
cells,  and  then  it 
saves  off  the 
different  results 
for  C! 
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Example:  Adding  Reality  to  Schedules-1 
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Process 

Durations 

Step 

Best 

Expected 

Worst 

1 

27 

30 

75 

2 

45 

50 

125 

3 

72 

80 

200 

4 

45 

50 

125 

5 

81 

90 

225 

6 

23 

25 

63 

7 

32 

35 

88 

8 

41 

45 

113 

9 

63 

70 

175 

10 

23 

25 

63  _ f  Wl 

lat  would  you  forecast  the 
hedule  duration  to  be? 

500  ' 

\=Z -  sc 

\  \ 
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Adding  Reality  to  Schedules-2 


The  project  is  almost 
guaranteed  to  miss 
the  500  days  duration 
1 00%  of  the  time. 


^  (-Infinity 


t  Forecast  Total  Duration 

Edit  View  Forecast  Preferences  Help 

1.000.000  Trials 

Frequency  View 

995,599  Displayed 

Total  Duration 


Certainty:  j 90.0960 


19,000 

18,000 

17,000 

16,000 

15,000 


With  90%  confidence, 
the  project  will  be 
under  817  days 
duration. 


9,000  ^ 


D.00  680.00  700.00  720.00  740.00  760.00  780.00  800.00  820.00  840.00 

Days 


.00  900.00 


4  ! 
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Adding  Reality  to  Schedules-3 


>  Forecast  Total  Duration 

Edit  View  Forecast  Preferences  Help 

1.000.000  Trials 

Frequency  View 

995.599  Displayed 

Total  Duration 


With  only  50% 
confidence,  the  project 
will  be  under  731  days 
duration. 


19,000 
18,000 
-  17,000 

16,000 


9,000  Q 


560.00  580.00  600.00  620.00  640.00  660.00 


0.00  700.00  720.00  740.00  760.00  780.00  800.00  820.00  840.00 

Days 


.00  900.00 


►  (-Infinity 


Certainty:  1 50.0635 


<j  m 
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Example:  BBN  Quality  Model 


Testing  Quality 


0.43 

0.4 

0.32 

0.24 

0.1G 

0.03 

0.0 


Very  Low 


High 


Very  High 


X 


Defects  in  Product 

0.4S  _ 

0.4  _ 

0.32  _ 

0.24  _ 

Q.16  _ 

0.08  _ 

0.0 

-rflTrrrTTTTrr 

L. 

llllll 

bbbbbri?J“' 

cncncncntn-i-i-i 
~j  -j  ^  ^  ^  -j  -j  -j 

-j 

b 

-i 

|*rob  of  Finding  Defect  in  Tesj 


7.0 

6.0 

G.O 

4.0 

3.0 

2.0 


■rmfFrn 


I  I  I  I 

oooooooooo 
b  2.  ho  gj  L.  cn  It.  -.j 


z 


Defects  Found  During  Test 


1.0 

0.9 

0.6 

0.7 

0.G 

0.5 

0.4 

0.3 

0.2 


Predict  the  probability  of 
finding  a  defect  during  a 
test  by  learning  what  the 
quality  of  testing  is. 

Predict  defects  found  by 
learning  more  about  the 
expected  incoming 
defect  level  and  the 
ability  to  find  defects 
with  testing. 


AgenaRisk.  http://www.agena.co.uk. 
(URL  valid  as  of  April  2007) 
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Example:  Discrete  Event  Process  Simulation 


Grade  2 


Grade  3 


Each  activity,  such  as 
“staging”,  is  assigned 
information  about 
capacity,  time  to 
perform,  and 
information  about  input 
Vcjueue  lengths. 


Operation  Criteria 

1  Grade  1  items  cannot  be  mixed  with  Grades  2  and/or  3  durirfcTst aging  or  at  processing  stations 

2.  Grades  2  and  3  may  be  mixed  during  staging  and  at  processing  stations 

3.  Grade  1  items  can  only  be  processed  by  Stations  1  and  2 


iH 


2U 


Adapted  from  ProcessModel,  Inc. 
ProcessModel. 

http://www.processmodel.com. 

(URL  valid  as  of  April  2007) 


General  |  Batching  j  -cfion  |  Coat  |  Shift  |  Submodel  | 
|  Staging 


Name: 

Capacity:  jam 
Time:  |l 


input  Queue  Cap.:  555 


&  Stats  on  Output  Queue  Cap.:  1 55555 
|min  T  |  Object  type: 


Undo 

Help 


|  Activity 
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Step  1  -  Implement  the  Model  in  a  Tool 


SSTC  2010 


Statistical  Modeling:  Example  tools  include  Minitab,  SAS 
JMP 


Monte  Carlo  Simulation:  Example  tools  include  Crystal  Ball 
and  @Risk 


Probabilistic  Modeling:  Example  tools  include  AgenaRisk, 
Netica,  Hugin 


Discrete  Event  Simulation:  Example  tools  include 
ProcessModel  and  Savvion 
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Step  1  -  Example  Statistical  Package  Tools 
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http :  //ww  w .  minitab .  com/ 


Minitab^ 


Store  |  Contact  Us  |  Site  Map 


Products  Training  Support  Company  Reso 


Software  Solutions 
for  Quality  Professionals 


Select  Country 

|  USA  _  v  | 


Minitab5 15  Statistical  Software 

All  the  tools  you  need  to  analyze  your  quality  improvement  data. 
More  |  Tour  j  Try  |  Buy 


Customer  Center 
Minitab  for  Education 
Free  Web  Events 
Quality  Trainer  Login 


Quality  Companion5  3 

Everything  you  need  to  manage  your  improvement  projects  in  one  application. 
More  |  Tour  |  Try  (  Buy 


http :  //w  ww .  jmp .  com  / 


jmp 


Free  Trial  ;  Buy  JMP  ;  International  ;  About  Us  ;  Contact  ;  News  an<J  Events  ;  Blog  ;  Resource  Center 


HOME  SOFTWARE  INDUSTRIES  APPLICATION  AREAS  ACADEMIC  TRAINING 


SUPPORT  USERC 


READY  FOR  BLASTOFF? 

What  are  you  waiting  for?  Pick  a  Mission  Specialist  and  discover  hbw 
»  and  new  ways  to  explore  your  universe  with  JMP®  8. 


¥ 


The  Rocket  Scientist 

Protecting  the  JMP  Dome  from  asteroids  while  controlling  costs. 


■V" 


NEWS 

AND 

EVENTS 

■  :ivd 

Predictive 

Analytics 


SOFTWARE 


JMP®  Statistical 
Discovery  Software 


JMP 

8 


Interactive.  Comprehensive. 
Highly  visual.  That  just 
begins  to  describe  JMP 
statistical  discovery 
software.  It’s  the  SAS 


nrnrlur.t  that  rlvnamir.allv 
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Step  1  -  Example  Monte  Carlo  Simulation  Tools 


http :  //www .  palisade .  com/ 


SSPALISADE  Maker  at  the  world’s  leading  risk  and  decision  analysts  software  ^  I  free  trials  |  cone 


Integrated  Decision  Making  in  Excell 


PRODUCTS  CUSTOMERS  TRAINING  ft  CONSULTING  SUPPORT 


EVENTS  COMPANY 


DecisionTools 

Suite 

Complete  Risk  and 
Derision  Analysis  Toolkit 

@RISK 


Headlines 


@RISK  Used  to  Simulate 
Consequences  of  Subprime 
Mortgage  Crisis 
Monmouth  University's  Dr.  Roy 


Conferences 


http :  //ww  w .  orade .  com/cry  stalball/index .  htrnl 


PRODUCTS  AND  SERVICES  i  INDUSTRIES  i  SUPPORT  i  PARTNERS  I  COMMUNITIES 


ORACLE  CORPORATION 

ABOUT  ORACLE 


About  Orade  He  me 
Acquisitions 
Analyst  Relations 


Oracle  and  Crystal  Ball 


As  a  result  of  its  acquisition  of  Hyporion,  Oraclo 


Beard  of  Directors 
Co  nta  ct  Ora  de 
Corporate  Go verna  n  ce 


Crystal  Ball  software  is  a  leading  spreadsheet-t 
500,  Crystal  Ball  is  used  by  customers  from  a  bi 
schools  worldwide  for  teaching  risk  analysis  coi 


Customers 
Exe  cuti  ves 
H  i  story 


The  diverse  applications  for  Crystal  Ball  include 
Management,  you  can  apply  the  power  of  Crysta 


Software  Engineering  Institute  CarnegieMellon 


Robert  Stoddard  and  Dave  Zubrow 

©2010  Carnegie  Mellon  University 


58 


SSTC  2010 


Step  1  -  Example  Probabilistic  Modeling  Tools 


“AGENARISK”  http://www.aqena.co.uk/ 

a  •  ena 

_ J  Bayesian  Met  work  and  Simulation;  Software  for  Risk  Analysis  and  Decision  Support 


HOME  PRODUCTS  CASE  STUDIES  RESOURCES  SERVICES  HEWS  ABOUT  US 


Latest  News  /  Articles 


You  are  here:  Home  » 
Last  Updated:  01/02/2007 


Bayesian  models  used  to 
reduce  drug  development 
costs  by  $283  million  per 
approved  drug 


Bayesian  nets  provide  radical 
improvements  in  software 
defect  prediction 

Avoiding  legal  errors  with 
simple  Bayesian  reasoning 


“NETICA”  http://www.norsvs.com/ 

HNorsys 

SOFTWARE  CORR 


Products  Downloads  Resources  Site  Map  Order 


NORSYS  makes  advanced  Bayesian  belief 
network  and  influence  diagram  technology 
practical  and  affordable. 


“HUGIN”  httD://www.huain.comf/ 

HUGvN 


riTACT  SITEMAP  SEARCH 


El 


EXPERT 


PRO  DUCTS/ SERVICE  3  j  DEVE LG P E R | C A 5 E  STUD  I  ES  |  NEW  5  |  P  ARTN  E  R  j  CO  M  P  A  N  Y  INFO 


”...  a  world  leader  in  developing  tech 

for  advanced  knowledge  nflhagement 


Hugin  Expert's  unique  technology  enables 
you  to  create  intelligent  products  and  1 
services  based  on  criteria  such  as  speed, 
precision,  robustness  and  ease-of-use 


Hugin  Training  Course 


Our  Hugin  courses  in  Bayesian 
networks,  have  now  been  scheduled  for 
2007.  Join  our  next  training  course  in 
Copenhagen  scheduled  for  February 
27th -March  1  st. 

::more:: 

NEWS  ITEM 


^  H)  ::  Seminar : 
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Step  1  -  Example  Discrete  Event  Simulation 
Tools 

http://www.processmodel.com 


3  ProcessModel,  Inc.  -  Business  Process  Improvement  Solutions  -  Microsoft  Internet  Explorer 


File  Edit  View  Favorites  Tools  Help 


G***  •  «,  |  I  ,  *«*,  fr**—  €  :  •  - 


Address  |£j  http  :  //www .  processmodel .  com/ 


Y/  ’  &A 

m  ProcessModel,  Inc.  -  Business  Pr. . .  I  +  Add  Tab 


|  Search  Web  -  $  0  j  OMa'1  T  My  Yahoo!  -  0  Shopping  »  Q,  Football 


http://www.savvion.com 


|1  Sawion  December  ZOO 6  Newsletter  -  Microsoft  Internet  Explorer 

File  Edit  View  Favorites  Tools 

Help 

QBack-  ^  \$\  '> 

P»*k  -^Favorites  &  .  .  ^  &  *  Q  <  1&r 

Address  http :  //www .  sawion . com/newsletter /dec_2006 .  html 

Y'  *  *-r 

^  |  [  Search  Web  -  ...  '  ^  |  OMail  -  ^  My  Yahoo!  -  ,;'ij  She 

|G|  Savvion  December  Z006  Newsle...  +  Add  Tab 

1  About  Us  I  Products 

|  Solutions  1  Customers  |  industries  |  Literature  |  News 

SAV  yiON 

B  nOMMFP.T 

1-533-544-5-511  l  Having  trouble  loading  hiib  e-maiP  Read  rt  o.ihna  Add  nemalalter  ffaavvion.  com  to  your  IieI  of  sale  oontsi 


NOVEMBER/ 

DECEMBER2006 

IN  THIS  ISSUE 

Emerging  T rends  in 

BP  P.1:  What  Happened 

in  2006,  and  What's 

Ahead  in  2007 

**  Sawion 

BusinessPJanager  7.0 

is  Here! 

►  *  Successful  Innovation 
Road s how  Offers  BPM 

Proof  Points 
**  Introducing  the  First 


Emerging  Trends  in  BPM: 

What  Happened  in  2006,  and  What's 
ahead  in  2007 


IBMIflil 


Guest  contributor  BPM  consultant 
and  blogger  Sandy  Kemsley 
shares  an  insider's  look  at  the  past 
year,  and  what  we  can  expect  in  the 
year  to  come. 

The  BPM  market  continues  to 
evolve,  and  although  2006  has  seen 
some  major  events,  there  will  be 
even  more  in  2007.  This  column 
takes  a  high-level  view  of  four  areas 
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Step  2  -  Create  Predictions  with  Both 
Confidence  and  Prediction  lntervals-1 


SSTC2010 


Because  the  central  theme  of  CMMI  High  Maturity  is  understanding  and 
controlling  variation,  PPMs  produce  statistical  intervals  of  behavior  for 
outcomes  such  that  individual  predicted  values  will  have  an  associated 
confidence  level 


All  of  the  Process  Performance  models  discussed  provide  the  ability  to 
compute  both  the  confidence  and  prediction  intervals  of  the  outcomes. 
These  intervals  are  defined  on  the  next  slide 
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SSTC  2010 

Step  2  -  Create  Predictions  with  Both 
Confidence  and  Prediction  lntervals-2 

Confidence  Intervals:  The  statistical  range  of  behavior  of  a 
an  average  value  computed  from  a  sample  of  future  data 
points 


Prediction  Intervals:  The  statistical  range  of  behavior  of 
individual  future  data  points 


Note:  Prediction  Intervals  are  almost  always  much  wider  than 
confidence  intervals  because  averages  don’t  experience  the  wide 
swings  that  individual  data  points  can  experience  (similar  to  how 
individual  grades  in  college  compared  to  your  grade  point  average) 
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Step  3  -  Validating  and  Maintaining  PPMs  - 1 


SSTC  2010 


Initial  estimation  of  a  PPM  typically  yields 

•  Equation  or  function  describing  the  relationship  between 
independent  variables  (x’s)  and  the  dependent  variable  (y) 

•  An  indication  of  the  goodness-of-fit  of  the  model  to  the  data  (e.g., 
R-square,  Chi-square) 

These  do  not  necessarily  indicate  whether  the  model 
provides  sufficient  practical  value 

•  Track  and  compare  predictions  with  actual  results 

•  Failure  to  meet  business  criteria  (e.g.,  +/- 10%)  indicates  need  to 
recalibrate  (i.e,  same  variables  with  different  data)  or  remodel  (new 
variables  and  data) 
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Step  3  -  Validating  and  Maintaining  PPMs  -  2 


One  strategy  to  jump  start  this  process  is  to  use  half  the 
data  to  estimate  the  model  and  the  other  half  for  validation 
(and  other  variations  on  this  theme) 


A  second  strategy  is  to  accept  that  some  period  of  time 
going  forward  will  be  needed  to  collect  sample  data  by 
which  to  validate  the  PPM 
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Step  4  -  Confirm  the  PPM  Meets  the  Healthy 
Ingredients 

PPMs  can  have  the  greatest  business  benefit  when  they 
meet  all  of  the  healthy  ingredients 


However,  PPMs  should  not  only  be  evaluated  in  isolation, 
but  rather,  as  a  collection  of  models  enabling  the 
organization  and  it’s  projects  to  most  likely  exhibit  superior 
results 


That  said,  not  every  PPM  has  to  exhibit  each  and  every 
healthy  ingredient  to  be  considered  as  a  member  of  the 
portfolio  of  PPMs  serving  the  organization. 
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Tips  -  Barriers  to  Building  PPMs 


SSTC  2010 


Lack  of  compelling  outcomes  to  predict  due  to  misalignment 
with  critical  business  goals,  usually  caused  by  insufficient 
management  sponsorship  and  involvement 

Lack  of  a  connection  to  a  work  process  or  sub-process  such 
that  direct  changes  in  that  process  or  sub-process  can  help 
cause  changes  in  predicted  outcomes 

Insufficient  process  and  domain  knowledge  which  is 
necessary  to  identify  the  probable  x  factors  to  predict  the 
outcome 

Insufficient  training  and  practice  with  modeling  techniques 
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Tips  -  Documentation  Needed  when  Building 
PPMs-1 

Similar  to  the  existing  SEI  Indicator  Template  but  with  some 
additional  information  content: 

i. Identity  of  associated  processes  and  subprocesses 

2.ldentity  of  the  outcome  measure  (y)  and  the  x  factors 

3.Data  type  of  all  outcome  (y)  and  x  factors 

^Statistical  evidence  that  the  x  factors  are  significant  (e.g.  p 
values  of  individual  x  factors) 

s.Statistical  evidence  of  the  strength  of  the  model  (e.g.  the 
adjusted  R-squared  value) 

6. The  actual  prediction  equation  for  the  outcome  (y) 

7. The  performance  baselines  of  the  x  factors 


Software  Engineering  Institute 


Carnegie  Mellon 


Robert  Stoddard  and  Dave  Zubrow 

©2010  Carnegie  Mellon  University 


67 


SSTC  2010 


Tips  -  Documentation  Needed  when  Building 
PPMs-2 


Similar  to  the  existing  SEI  Indicator  Template  but  with  some 
additional  information  content  (continued): 

s.The  resulting  confidence  interval  of  the  predicted  outcome 

9. The  resulting  prediction  interval  of  the  predicted  outcome 

10. Use  case  scenarios  of  how  the  PPM  is  intended  to  be  used 
by  different  audiences  for  specific  decisions 

11.  Description  of  how  often  the  PPM  is  updated,  validated, 
and  calibrated 


^.Description  of  how  often  the  PPM  is  used  to  make 
predictions  with  results  shown  to  decision-makers 


^.Description  of  which  organizational  segment  of  projects  the 
PPM  applies  to 
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Overview  of  the  Steps 
to  Build  PPMs 

-  Usinq  PPMs 
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Using  PPMs 


SSTC  2010 


Use  these  models  to  assist  with  statistical  management  of 
critical  subprocesses 


Use  the  predictions  of  these  models  to  make  decisions  and 
take  preventive  and  mitigative  action 


Use  these  models  to  help  with  CAR  and  OID 


Coach  audiences  on  how  to  understand,  interpret  and  draw 
conclusions  from  process  performance  models 


Software  Engineering  Institute 


Carnegie  Mellon 


Robert  Stoddard  and  Dave  Zubrow 

©2010  Carnegie  Mellon  University 


70 


SSTC  2010 

Take  Action  Based  on  Results  of  PPM 
Predictions 

If  a  PPM  model  predicts  an  unacceptable  range  of  values 
for  a  particular  outcome,  then  early  action  can  influence  a 
more  desirable  range  of  outcome 


Once  a  PPM  model  predicts  a  range  of  values  for  a 
particular  outcome,  then  actual  values  can  be  compared  to 
the  range.  If  the  actual  values  fall  outside  the  range,  it  may 
be  treated  similarly  to  a  point  on  a  control  chart  falling 
outside  of  the  control  limits 


Use  PPM  predictions  to  help  inform  process  composition 
decisions  so  that  business  goals  may  be  optimized 
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How  PPMs  Assist  CAR 
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•  Aid  impact,  benefit,  and  ROI  predictions  for 

•  Selecting  defects  for  analysis 

•  Selecting  action  proposals  for  implementation 

•  Use  PPMs  to  identify  potential  sources  of  the  problem  or 
defect 

•  Use  PPMs  to  understand  the  interactions  among  selected 
improvements;  and  the  combined  predicted  impacts,  costs, 
and  benefits  of  the  improvements  (considered  as  a  set) 

•  Compare  the  result  versus  the  original  PPM-based  prediction 
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How  PPMs  Assist  OID 


SSTC  2010 


•  Select  process  improvement  proposals  for  implementation 
by  aiding  impact,  benefit,  and  ROI  predictions 

•  Identify  opportunities  for  improvement 

•  Use  PPMs  to  understand  the  interactions  among  selected 
improvements;  and  the  combined  predicted  impacts,  costs, 
and  benefits  of  the  improvements  (considered  as  a  set) 

•  Prioritize  improvements  based  on  ROI,  cost,  risk,  etc. 

•  Confirm  the  prediction  (provides  input  to  maintaining  PPMs) 
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What  is  Sub-optimization  and  how  can  PPMs 
help? 

Sub-optimization  is  where  one  parameter  is  optimized  at  the 
expense  of  other(s) 

•  Reduce  delivered  defects,  but  are  late  and  over  budget 

•  Meet  the  cost  goal  but  don’t  deliver  desired  functionality 

PPMs  allow  you  to 

•  Gage  the  trade-offs  amongst  multiple  goals 

•  Gage  the  effects  of  changes  to  multiple  parameters 
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PPM  Exercise  1 : 
Constructing  a  Product 
Business  Case  with 
Monte  Carlo  Simulation 
and  Optimization 
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Monte  Carlo  Simulation  Steps  with  Crystal  Ball 
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Crystal  Ball  Toolbar 
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Define  decision 

You  can 

(Lets  you  identify  a  cell 

copv,  paste 

as  a  decision  cell  to  be 

and  dear 

used  in  Optimization 

Crystal  Ball 

Modeling) 

identities  to 

save  time 

ijC^Copy  L-fj  Select  ¥ 

.  jF.vr  jj§|  Freeze 
Define  Define  Define  p 

Assumption  T  Decision  For^ast  Clear  Cell  Prefs 


m 


Define  forecast 
(Lets  you  identify  a  cell  as 
an  outoome  that  you  want 
to  study) 


Define  assumption 
(Lets  you  identify  a  cell 
as  an  uncertain  cell  with 
a  distribution) 


Start  simulation 

(Start  simulation 
once  all  settings 
are  made) 


Start 


Single  step 
(Lets  you  run  the 
simulation  step  by  step. 
Normally  used  to 
debug  issues  with 
simulation) 


Create  report 

(Creates 
standardized 
reports  of  the 
simulation) 


Step 


Tools  T 
□  Save  or  Restore  T 
FyL  Run  Preferences 
Run 


LS 


OptQuest 


Reset  simulation 
(Restart  simulation 
and  erase  previous 
results) 


Stop  simulation 

(You  can  stop 
the  simulation 
midstream) 


View  Create  Extract 
Iharts  Report^  Data 

Analyze 


Qptquest 

(Begin 

optimization) 


Run  preferences 
(Enables  the  settings  of  how 
long  the  simulation  runs,  etc. . .) 


©Help 
||_  Resources 
About 
Help 


Extract  data 

(Allows  the 
capture  and 
saving  of  the 
actual  simulation 
data  from  all  the 
runs) 
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Benefits  of  Using  Optimization  Modeling 
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Monte  Carlo  simulation  models  can  only  provide  a  range  of  possible 
outcomes  for  any  situation.  They  do  not  identify  ways  to  control  the 
situation  to  achieve  the  best  outcome. 


Optimization  modeling 

•  automates  tens  of  thousands  of  decision  “what-ifs”  from  a  Monte  Carlo 
simulation  to  determine  the  best  possible  solution 

•  is  easy  to  use,  not  tedious  and  time  consuming  like  many  other 
analytical  methods 

•  uses  state-of-the-art  algorithms  for  confidently  finding  optimal  solutions 

•  supports  decision  making  in  situations  where  significant  resources, 
costs,  or  revenues  are  at  stake 
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Steps  for  Optimization  Using  Crystal  Ball 
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2 

Business  Case  Example  for  Feature  Inclusion  Decision 

3 

in  Upcoming  Hospital  Records  Software  Project 

4 

Develop 

Feature? 

/ 

\ 

- 1 — i - 1 

Business  Case  Monte  Carlo  Optimizatior 

Feature  ID 

Feature  Description  \ 

5 

1 

1 

Online  Web  Access  \ 

6 

1 

2 

Real-time  Updating  of  Information 

7 

0 

3 

Shared  User  Information 

8 

0 

4 

Report  Historical  Usage 

9 

0 

5 

Conduct  Security  Check 

IQ 

0 

6 

Confirm  Transactions 

11 

0 

7 

Cross  Check  Different  Patients  Information 

12 

0 

8 

Trace  Prescriptions  Used 

13 

0 

9 

Trace  Assigned  Doctor 

14 

0 

10 

Trace  Hospital 

15 

1 

11 

Conduct  Periodic  Audit 

16 

1 

12 

Check  for  Corrupt  Data 

17 

0 

13 

Provide  Conflict  Warning 

18 

0 

14 

Identify  Incomplete  Records 

19 

1 

15 

Compute  Cycle  Times  on  Value  Stream 

20 

1 

16 

Enable  cross  hospital  sharing  of  data 

21 

0 

17 

Provide  Security  Encryption  for  Sensitive  Data 

22 

1 

18 

Enable  workflow  automation  messages 

23 

1 

19 

Require  peer  review  of  critical  data  inputs 

24 

1 

20 

Provide  for  automated  archival  of  information 
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Minimum 

Expected  Budget 
Needed  ($K) 

Most  Likely 
Expected  Budget 
Needed  ($K) 

Maximum 

Expected 
Budget  Needed 
($K) 

Simulated 
Budget  ($K) 

Budget  Used  in 

Simulation 

Feature 

Scenario  ($K) 

$10,000 

$12,000 

$29,260 

$0,000 

$0,000 

$12,000 

$14,400 

$21,870 

$0,000 

$0,000 

$13,540 

$16,248 

$27,420 

$0,000 

$0,000 

$11,298 

$13,558 

$19,880 

$0,000 

$0,000 

$25,000 

$30,000 

$35,290 

$0,000 

$0,000 

$21,430 

$25,716 

$29,830 

$0,000 

$0,000 

$19,450 

$23,340 

$39,750 

$0,000 

$0,000 

$18,390 

$22,068 

$38,234 

$0,000 

$0,000 

$17,420 

$20,904 

$29,774 

$0,000 

$0,000 

$29,170 

$35,004 

$51,960 

$0,000 

$0,000 

$26,290 

$31,548 

$62,948 

$0,000 

$0,000 

$21,290 

$25,548 

$39,497 

$0,000 

$0,000 

$21,990 

$26,388 

$34,659 

$0,000 

$0,000 

$27,990 

$33,588 

$39,774 

$0,000 

$0,000 

$39,230 

$47,076 

$57,849 

$0,000 

$0,000 

$41,090 

$49,308 

$72,895 

$0,000 

$0,000 

$38,210 

$45,852 

$67,391 

$0,000 

$0,000 

$0,000 

$31,280 

$37,536 

$47,324 

$0,000 

$31,670 

$38,004 

$49,846 

$0,000 

$0,000 

$27,720 

$33,264 

$39,888 

$0,000 

$0,000 

$581,350 

Total  Budget  »>»»»»»»» 

$0,000 
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SSTC  2010 


A  Minimum 

Expected 
Calendar  Days 
Needed 


15 


10 


12 


25 


30 


26 


18 


15 


19 


28 


22 


26 


33 


18 


17 


26 


21 


22 


26 


Most  Likely 
Expected 
Calendar  Days 
Needed 


Maximum 
Expected 
Calendar  Days 
Needed 


20 


18 


15 


38 


38 


30 


29 


22 


26 


14 


37 


29 


40 


40 


26 


21 


29 


35 


29 


30 


30 


36 


56 


49 


75 


48 


62 


36 


39 


31 


45 


49 


67 


78 


40 


38 


37 


59 


51 


47 


Simulated 
Calendar  Days 


Total  Calendar  Days  »»»»»»» 


Calendar  Days 
used  in 
Simulation 
Feature 
Scenario 


0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 
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Expected  Senior 

Resource 

Needed 

Actual  Senior 

Resource  Used 

in  Simulation 

Relative 

Customer 

Value 

Customer  Value 

in  Simulation 

0 

0 

1.00 

1.00 

0.2 

0.2 

2.00 

2.00 

0.3 

0 

1.20 

0.00 

0 

0 

1.50 

0.00 

0.12 

0 

1.80 

0.00 

0.15 

0 

0.90 

0.00 

0.19 

0 

0.30 

0.00 

0.25 

0 

0.80 

0.00 

0 

0 

1.70 

0.00 

0 

0 

1.20 

0.00 

0 

0 

1.90 

1.90 

0.65 

0.65 

2.40 

2.40 

0.34 

0 

2.70 

0.00 

0.29 

0 

3.00 

0.00 

0.21 

0.21 

2.20 

2.20 

0.17 

0.17 

1.70 

1.70 

0 

0 

1.95 

0.00 

0 

0 

2.67 

2.67 

0 

0 

4.00 

4.00 

0 

0 

2.39 

2.39 

Total  Resource» 

1.23 

Value»> 

20.26 

~r 
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SSTC  2010 


Home 


isert  Page  Layout  Formulas  Data  Review  View  Add-Ins  Crystal  Ball 


Define! 

Define  ■ 

Assumptioi\ 

Decision  « 

sfine 

jrecast 


j^Copy  L^Select”' 
:'aste  i  Freeze 
Clear  &  Cell  Prefs 


l 

C  i 
%J 

i 

\  j&Tools- 

■_J  Save  or  Restore  T 

c> 

JBi-Mi 

„  li 

Start  Stop  Reset 

steP  Run  preferenCe5 

OptQuest 

View 

Charts 

Run 

Define  Decision  Variable 

Define  the  selected  cells  as  decision  (or 
control]  variables  in  your  spreadsheet 
model. 


Hospital  Records  Software  Project 


Crystal  Ball 

Press  FI  for  more  help. 


Highlight  Cell  A22  and  then  hit 
the  Define  Decision  icon 


3  Shared  User  Information 

4  Report  Historical  Usage 

5  Conduct  Security  Check 

6  Confirm  Transactions 


Minimum 

Expected  Budget 
Needed  ($K) 


$10,000 

$12,000 

$13,540 

$11,298 

$25,000 

$21,430 


1  1 


t\ 


fin  c  n 
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SSTC  2010 


Define  Decision  Variable:  Cell  All 


Highlight  Cell  A23  next  and 
then  hit  the  Define 
Decision  icon 


Name: 

Bounds 


Feature  1 S  Decision 


Lower: 


0.00 


% 


Upper: 


1.00 


=k. 

— 


— ■ 


Type 

Continuous 


(.!*)  Discrete 


Step: 


1.00 


=W. 


OK 

j  Cancel 

Help 
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SSTC  2010 


Define  Decision  Variable:  Cell  A23 


Highlight  Cell  A24  next  and 
then  hit  the  Define 
Decision  icon 


Name: 

Bounds 


Feature  1 9  Decision 


Lower: 


VI  ¥ 


E3RIS1 

%sl  Upper: 

1.00 

Type 

Continuous 

®  Discrete 


Step: 


1.00 


— 


<a 


OK 

}  Cancel 

Help 
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SSTC  2010 


Define  Decision  Variable:  Cell  A 24 


Name: 


Feature  20  Decision 


v 


Bounds 


Lower: 


0.00 


Upper: 


1.00 


31.1 


Type 

Continuous 


(3)  Discrete 


Step: 


1.00 


^.1 


OK 


32 


Cancel 


Help 
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SSTC2010 


Business  Case  Monte  Carlo  Optimization- 

Home  Insert  Page  Layout  Formulas  Data  Review  View  Add-Ins  Crystal  Ball 


Copy 

Define  Define 

sion  Forecast  ^  ^  ear 

fine  _ 


Select  T 
Freeze 

m  Cell  Prefs 


k  □ 


"J^Tools  T 


Start 


Step 


Reset 


Step 


Define  Assumption 

Define  the  selected  cells  as  assumptior 
by  choosing  from  a  gallery  of 
probability  distributions  types, 

Assumptions  are  the  uncertain  variables 
in  your  spreadsheet  model. 


Crystal  Ball 

Press  FI  for  more  help. 


Highlight  Cell  M22  next 
and  then  hit  the  Define 
Assumption  icon 


7 

8 
9 

in 


o 

0 

0 

n 


3 

4 

5 

f, 


Save  or  Restore  T 
Q  Run  Preferences 
Run 


4 


OptQu 


e  Project 


Feature  Description 


Mi 

Expec 

Nee 


Online  Web  Access 
Real-time  Updating  of  Information 
Shared  User  Information 
Report  Historical  Usage 
Conduct  Security  Check 

r nnfirm  Transartinns 
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SSTC  2010 


Define  Assumption:  Cell  M22 


Edit  View  Parameters  Preferences  Help 


Name: 


Feature  1 8  Budget 


1 


Highlight  Cell  M23  next 
and  then  hit  the  Define 
Assumption  icon 


Triangular  Distribution 


$32,000  $34,000  $36,000  $38,000  $40,000  $42,000  $44,000  $46,000 
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Define  Assumption:  Cell  M23 


Edit  View  Parameters  Preferences  Help 


Name:  Feature  19  Budget 


Highlight  Cell  M24  next 
and  then  hit  the  Define 
Assumption  icon 


Triangular  Distribution 


$32,000  $34,000  $36,000  $38,000  $40,000  $42,000  $44,000  $46,000  $48,000  $50,000 


Minimum 


%  Likeliest 

$38,004 

SI  Maximum 

$43,848 

=W. 

— 


OK 

l  Cancel 

Enter 

Gallery 

Correlate... 

Help 
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SSTC  2010 


A 


J  lS  a  v* 


Business  Case  Monte  Carlo  ( 


Home 


Insert 


Define  Define 
Assumption  T  Decision 


Formulas 


Data 


Review 


View 


Add-Ins 


Crystal  I 


Select  T 
jj^l  Freeze 
Cell  Prefs 


ii> 

Start  Stop  Reset  Step 


Define  Forecast 

Define  the  selected  cells  as  output 
variables  of  interest  in  your 
spreadsheet  model, 

@  Crystal  Ball 

Press  n  for  more  help. 


Highlight  Cell  026  and 
then  hit  the  Define 
Forecast  icon 


Feature  Description 


5 

6 

7 

8 

q 


Tools  T 
y  Save  or  Res 
Run  P  ref  ere 


Run 


Online  Web  Access 

V 

Real-time  Updating  of  Information 

< 

V 

Shared  User  Information 

< 

V 

Report  Historical  Usage 

< 

V 

Conduct  Security  Check 

< 

Minim 

Expected  I 
Needed 
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Define  Forecast:  Cell  Y26 


Name  Total  Days 


vN 


Units: 


LSL: 


Target: 


M  USL: 


Forecast  Window  Precision  Filter  Auto  Extract 


View:  Frequency 


1  Split  view 


Window 


0  Show  automatically 

<*}  Wh i I e  running  simulation 
ij  When  simulation  stops 


Fit  distribution 

]  Fit  a  probability  distribution  to  the  forecast 


<c 


SSTC  2010 


Fit  Options. 


After  hitting  OK,  Highlight 
Cell  AC26  next  and  then 
hit  the  Define  Forecast 
icon 


Apply  To.. 


Defaults... 


Help 
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Define  Forecast:  Cell  AC26 


SSTC  2010 


Name  Total  Senior  Resource 
Units: 


m  A 


LSL: 

Target: 


=W: 

M  USL: * (*) 


Forecast  Window  Precision  Filter  Auto  Extract 


View: 


Frequency 


O  Split  view 

Window 

[^1  Show  automatically 

(*)  While  running  simulation 
O  When  simulation  stops 


Fit  distribution 

[]  Fit  a  probability  distribution  to  the  forecast 


Fit  Options... 


After  hitting  OK,  Highlight 
N  Cell  AG26  next  and  then 
hit  the  Define  Forecast 
icon 


OK 

j  Cancel 

Apply  To... 

Defaults. .. 

Help 
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SSTC  2010 


A 


□  ^  14  ^ 


Business  Case  Monte  Carlo  Optimization-vOlO  -  Micro 


Home 


Insert  Page  Layout  Formulas  Data  Review  View  Add-Ins  Crystal  Ball 


u. 


y>  kj, 


Define  Define  Define 
Assumption  T  Decision  Forecast 

Define 


w  c°py 

Paste 

&  Clear 


Select  T 
iji  Freeze 
m  Cell  Prefs 


3 

4 

Develop 

Feature? 

5 

1 

6 

1 

7 

0 

8 

0 

9 

0 

10 

0 

Start  Stop  Reset  Step 


Tools  T 

SI  Save  or  Restore 
S  Run  Preferences 
Run 


in  Upcoming  Hospital  Records  Sc 


Feature  ID 


OptQuest 

Search  for  and  find  optimal  solutions  to 
your  simulation  models, 

O  Crystal  Ball 

Press  FI  for  more  help, 


OptQuest 

1  — j 

■ 

1  View 

/lharts  F 

i 

Feature  Description 


Online  Web  Access 


Real-time  Updating  of  Information 


Shared  User  Information 


Report  Historical  Usage 


Conduct  Security  Check 


Confirm  Transactions 


Minimum 

Expected  Budget 
Needed  ($K) 


$10,000 


$12,000 


$13,540 


$11,298 


$25,000 


$21,430 
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SSTC2010 


Select  an  objective  and  optionally  specify  requirements 

Primary  workbook: 

Business  Case  Monte  Carlo  Optimi;  v 

Objectives:  ^ 

Exclude 

Maximize  thp  5%  Percentile  of  Total  Customer  Value 

□ 

Minimize  thp  95%  Percentile  of  Total  Budget 

0 

Hit  the  Add  Objective  button  to  enter  the  first 
objective  seen  on  this  screen. 


Exclude 


□ 


□ 


□ 


Add  Objective 


Add  Requirement 


Efficient  Frontier 


Import.. 


Delete 


Software  Engineering  Institute  GarnedeMellon 


Robert  Stoddard  and  Dave  Zubrow 

©2010  Carnegie  Mellon  University 


SSTC  2010 


Select  an  objective  and  optionally  specify  requirements 


- \ 


Then,  Hit  the  Add  Requirement  button  3  times  to  enter  the  three 

Requirements  seen  on  this  screen. 


Requirements: 


Exclude 


The 

95% 

Percentile 

of  Total  Senior  Resource  must  be  less  than  2.10 

□ 

The 

95% 

Percentile 

of  Total  Days  must  be  less  than  300.00 

□ 

The 

95% 

Percentile 

of  Total  Budget  must  be  less  than  5&400.0D0 

□ 

Add  Objective 


Efficient  Frontier 


Import.. 


Delete 
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SSTC  2010 


Welcome 

Objectives 


Decision  Variabi 


Constraints 

Options 


ispa 


Review  decision  variables  and  change  properties  as  necessary 

]  Show  cell  locations 

Decision  Variables 

Lower  EL. 

f  ase  Ca...  l\per  B .  Type  St...  Fre...  A 

Feature  1  Decision 

0.00 

I  100  \ 

1.00  Discrete  1.00  O 

Feature  1 0  Decision 

0.00 

/  0.00  \ 

1.00  Discrete  1.00  O 

Feature  1 1  Decision 

0.00| 

1  1.00  ] 

[  1.00  Discrete  1.00  Q 

Feature  12  Decision 

0.00 

1.00  1 

|  1.00  Discrete  1.00  Q 

Feature  1 3  Decision 

0.0( 

0.00 

|  1.00  Discrete  1.00  Q 

Feature  14  Decision 

0.0( 

0.00 

|  1.00  Discrete  1.00 

Feature  1 5  Decision 

0.0( 

1.00 

|  1.00  Discrete  1.00 

Feature  1 6  Decision 

0.01 

1.00 

Feature  1 7  Decision 

0.0( 

0.00^ 

i  Enter  0  for  each  < 

Feature  1 S  Decision 

o.ot 

1.00  \ 

the  Base  case 

Feature  1 9  Decision 

o.ot 

1.00  \ 

Feature  2  Decision 

o.ot 

1.00  ) 

values.  This  is  tf 

Feature  20  Decision 

0.00 

1.00 

Feature  3  Decision 

0.00 

\  0.00  I 

starting  solution 

Feature  4  Decision 

0.00 

I  0.00  / 

where  the 

Feature  5  Decision 

0.00 

\  0.00  / 

Feature  6  Decision 

0.00 

\  0.00  / 

optimization  will 

Feature  7  Decision 

0.00 

\  0.00  / 

k  begin  looking. 

Run 


Close 


Help 
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OptQuest 
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SSTC  2010 


Advanced  Options 


Optimization  control  (additional) 


^Enable  low- confide  nee  testing 


(Improves  optimization  time  by  stopping  simulations  early  if  the  soluti 
inferior  to  the  best  solution) 


@  Automatically  stop  arter 
non-improving  solutions 


500 


Help 
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SSTC  2010 


Control  Panel  -  QptQuest 


Run  Analyze  Help 

Optimization  complete 

iiimmmmmmimmmmmimmmmmii 

00:00  Time  (min:sec):  09:59  10:00 

|  imimimiiimmiiimimmiiiimnimmiml 

Trials  10:000  10,000 


6>  Uk 
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SSTC  2010 


Best  Solution:  Simulation  #  656 


Objectives 

Value 

Maximize  the  5%  Percentile  of  Total  Customer  Value 

20.54 

—  Requirements 

Value 

The  95%  Percentile  of  Total  Senior  Resource  must  b... 

1.57 

The  95%  Percentile  of  Total  Days  must  be  less  than  ... 

296.10 

The  95%  Percentile  of  Total  Budget  must  be  less  tha... 

$359,952 
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SSTC  2010 


—  Decision  Variables 

Value 

Feature  1  Decision 

0.00 

Feature  1 0  Decision 

Z 

1 

1.00 

Feature  1 1  Decision 

0.00 

Feature  12  Decision 

1.00 

Feature  13  Decision 

£ 

1.00 

Feature  14  Decision 

0.00 

Feature  15  Decision 

/ 

1.00 

Feature  1 6  Decision 

Ld 

1.00 

Feature  1 7  Decision 

£ 

1.00 

Feature  IS  Decision 

0.00 

Feature  1 9  Decision 

£ 

1.00 

Feature  2  Decision 

1.00 

Feature  20  Decision 

1.00 

Feature  3  Decision 

0.00 

Feature  4  Decision 

0.00 

Feature  5  Decision 

0.00 

Feature  6  Decision 

0.00 

Feature  7  Decision 

0.00 

Feature  S  Decision 

0.00 

Feature  9  Decision 

0.00 
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OptQuest  Results 


- 


Decision  Variables 


Value 
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I 


Crystal  Ball  Report  -  OptQuest 

Optimization  started  on  8/21/2009  at  9:20:35 
Optimization  stopped  on  8/21/2009  at  9:30:36 


SSTC  2010 


Run  preferences: 

Stochastic  optimization  (with  simulation) 


Low-confidence  testing  on 

Maximum  trials  per  simulation  10,000 

Monte  Carlo 
Random  seed 
Precision  control  on 

Confidence  level  95.00% 

Run  statistics: 

Total  optimization  time  (min:sec)  10:01 

Number  of  simulations  795 

Stopped  by 

Trials  limit  reached  406 

Precision  control  0 

Low-confidence  testing  389 

Infeasible  constraints  0 

Simulation/second  (average)  1 

Other  statistics: 

Number  of  infeasible  solutions  537 

Due  to  requirements  537 

Due  to  non-linear  constraints  0 
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SSTC  2010 


Summary: 

After  795  solutions  were  evaluated  in  10  minutes  and  1  second, 
the  5%  Percentile  of  Total  Customer  Value  was  improved  to  20.54 


PoriBfiMmtilvfln 


himfth  i 
frit  iqM^4 

B+iC  f-1|  IMrVH 


Objectives 


Best  Solution: 


Maximize  the  5%  Percentile  of  Total  Customer  Value 


20.54  Cell:  AG26 


Requirements 

The  95%  Percentile  of  Total  Senior  Resource  must  be 
less  than  2.10 

The  95%  Percentile  of  Total  Days  must  be  less  than 


300.00 


The  95%  Percentile  of  Total  Budget  must  be  less  than 
$400,000 


=  ,  Software  Engineering  Institute  CarnegieMellon 


1.57  Cell:  AC26 
296.10  Cell:  Y26 

$359,952  Cell:  026 
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Total  Senior  Resource 


lO.QDQ 
9.  DOS 
EflOO 
7.  DOC 
fi.OOO 
5  DQQ 
DOS 
3  m 

2  DOO 

T  000 
0 


Statistics: 


Forecast  values 


Trials 

Mean 

Median 

Mode 


=^=-  Software  Engineering  Institute 


10,000 

1.57 

1.57 

1.57 
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Probability 


SSTC  2010 


Total  Days 
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Frequency 


SSTC  2010 


*  Opt  Quest  Results 

View 


fait 


Analyze  ^preferences  Help 


✓  Best  Solution 


Solution  Analysis 


Best  Solution 


Performance  Chart 


■3J 

1  35.00 


■D 

r_j 

O 

Cl 

■-S 


30.00 

25.00 


o  20.00 

>  15.00 
E  10.00 

o 

"55 
o 

1  o.oo 


5.00 


it 

++ 


*  ** 


**  * 

*  ** 
w  **  - 


►* 


-t^Z 


**+* 


*  * 
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SSTC  2010 

795  Total  Solutions  Solution  Analysis  View  258  Displayed 


Objective 

—  Requirements 

|— 1  Decision  Va 

A 

Maximize  5%  Percentile 

95%  Percentile  <  Z  ID 

95%  Percentile <  300  DO 

95%  Percentile  <  #400.000 

Rank 

Solution# 

T  otal  Customer  Value 

Total  Senior  Resource 

Total  Days 

Total  Budqet 

Feature  1  Decisk 

1 

656 

20.54 

1.57 

296.10 

S359.952 

C 

t 

2 

772 

20.51 

1.23 

289.29 

$371,264 

C 

3 

469 

19.64 

1.65 

295.60 

$329,433 

C 

t 

4 

740 

T9. 64 

1.48 

296.89 

$236,605 

C 

5 

472 

19.59 

1.57 

287.43 

3325. 140 

1 

T 

6 

627 

19.39 

1.65 

293.52 

$299,642 

c 

T 

7 

768 

19.36 

1.48 

283.53 

S345.677 

c 

t 

8 

549 

19.31 

1.19 

282.13 

$292,331 

c 

9 

284 

19.09 

1.70 

294.06 

S286.844 

1 

10 

271 

19.04 

1.36 

283.31 

3316.331 

c 

f 

11 

673 

18.91 

0.83 

279.33 

3369.156 

[ 

r 

12 

368 

18.39 

1.32 

286.77 

3291.631 

c 

r 

13 

428 

18.84 

1.06 

272.43 

$316,042 

1 

t 

14 

4  94 

18.84 

1.40 

270.25 

$301,929 

c 

15 

263 

18.79 

1.55 

296.84 

3293.061 

1 

t 

16 

474 

18.79 

1.67 

297.46 

3323.196 

1 

v 

T 

< 

> 

Statistics :  t  -  Low-confidence  sol  ution  (va  I  ues  a  re  a  pprox. ) 


Minimum 

0.00 

0.00 

0.00 

30.000 

0.00 

Mean 

13.93 

1.06 

238.17 

3248.456 

0.35 

Maximum 

20.54 

2.06 

299.12 

3371.264 

1.00 

Std.  Dev. 

4.13 

0.45 

59.66 

370.477 

0.48 

Show  the  best 

o 


1 5 


o 


10 


solutions 
%  of  solutions 


(5)  All  feasible  solutions  (258) 
N ew  best  sol ut ions  (1 2} 


Include 

0  Feasible  solutions  (258) 
]]  Infeasible  solutions  (537) 
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Objectives  Best  Solution: 


Maximize  the  5%  Percentile  of  Total  Customer  Value 

20.01 

Cell:  AG26 

O 

Requirements 

The  95%  Percentile  of  Total  Senior  Resource  must  be 

less  than  1.00 

0.41 

Cell:  AC26 

The  95%  Percentile  of  Total  Days  must  be  less  than 

300.00 

293.86 

Cell:  Y26 

The  95%  Percentile  of  Total  Budget  must  be  less  than 
$400,000 

$351,343 

Cell:  026 
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Decision  variables 

Feature  1  Decision 
Feature  10  Decision 
Feature  11  Decision 
Feature  12  Decision 
Feature  13  Decision 
Feature  14  Decision 
Feature  15  Decision 
Feature  16  Decision 
Feature  17  Decision 
Feature  18  Decision 
Feature  19  Decision 
Feature  2  Decision 
Feature  20  Decision 
Feature  3  Decision 
Feature  4  Decision 
Feature  5  Decision 
Feature  6  Decision 
Feature  7  Decision 
Feature  8  Decision 
Feature  9  Decision 


0 

•/ 

£ 

£ 

13 
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Best  Solution: 

0.00 

Cell:  A5 

1.00 

Cell 

A14 

1.00 

Cell 

A15 

0.00 

Cell 

A16 

0.00 

Cell 

A17 

0.00 

Cell 

A18 

1.00 

Cell 

A19 

0.00 

Cell 

A20 

1.00 

Cell 

A21 

1.00 

Cell 

A22 

1.00 

Cell 

A23 

1.00 

Cell:  A6 

1.00 

Cell: 

:  A24 

0.00 

Cell:  A7 

0.00 

Cell:  A8 

0.00 

Cell:  A9 

0.00 

Cell 

A10 

0.00 

Cell 

All 

0.00 

Cell 

A12 

1.00 

Cell 

A13 
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PPM  Exercise  2: 
Scheduling  Projects 
with  Monte  Carlo 
Simulation  and 
Optimization 
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Scheduling  With  Monte  Carlo  Optimization-v010.xls  file 


SSTC2010 


Example  of  Using  Monte  Carlo  Simulation  and  Optimization  to  Make  Decisions 

Regarding  the  Work  Activities  and  Schedule  for  the  Hospital  Records  Information  System  (HRIS) 


(NOTE:  The  following  choices  are  independent  decisions] 

Choice  One 

Choice  Two 

Critical 

Path  Tasks 

Task  Description 

Minimum 

Days 

Most 

Likely 

Days 

Maximum 

Days 

Minimum 

Days 

Most 

Likely 

Days 

Maximum 

Days 

Decision 

Variable 

for  Choices 

Value 

Used  in 

Given 

Decision 

Scenario 

Simulation 

Value 

Choice  1 

Simulation 

Value 

Choice  2 

l 

Reqts  Development 

Traditional  Spec  Driven 

Prototype  with  Customer  First 

32 

40 

50 

70 

90 

120 

1 

0 

0  0 

2 

Architecture/Design 

All  New  Code 

Major  Reuse  of  Code 

55 

70 

90 

8 

10 

15 

2 

0 

0  0 

3 

Code 

All  New  Code 

Major  Reuse  of  Code 

43 

50 

62 

17 

20 

28 

2 

0 

0  0 

4 

Unit/Integration  Test 

Informally  Performed 

Formally  Performed 

100 

110 

125 

140 

150 

185 

1 

0 

0  0 

5 

Acceptance  Test 

Only  Choice 

19 

30 

39 

1 

0 

0 

Total  Critical  Path  Days  >J 

0 
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*  fSl  - 

ut  Formulas 


Data 


Scheduling  with  Monte  Carlo  Optimization-vOlO  -  Microsoft  Excel 

Review  View  Add-Ins  Crystal  Ball 


spy 

aste 

ear 


Select  T 
1^  Freeze 
m  Cell  Prefs 


Start  Stop  Reset  Step 


(  Tools  -  / 

'jJ  Save  or  Restora  T 
P^L  Run  Preferences 

<•> 

OptQuest 

K 

\  ^  I  XT] 

V  - 1  * r 

1  ULhP 

1  iew  Create  E  rract 

/  Report  ^  D.?-- 

ia< 

o 

Run  \ 

k. 

'  Analyze 

(NOTE:  The  following  chose 

Choice  One 

Minimum 

Days 

Most 

Likely 

Days 

Maximum 

Days 

Traditional  Spec  Driven 

32 

40 

50 

All  New  Code 

55  70  90 

All  New  Code 

43 

50 

62 

Informally  Performed 

100  110  125 

OptQuest 

Search  for  and  find  optimal  solutions  to 
your  simulation  models, 


O  Crystal  Ball 

Press  FI  for  more  help. 


Days 


Days 


Days 


ision 
iable 

tor  choices  Scenario 


Value 
Used  in 

Given  Simulation 
Decision  Value 
Choice  1 


Simulation 
Value 
Choice  2 


Prototype  with  Customer  First 
70  90  120 

Major  Reuse  of  Code 
8  10  15 

Major  Reuse  of  Code 
17  20  28 

Formally  Performed 
140  150  185 


0 
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Welcome 
Objectives 
Decision  Variabl 


Constraints 


Options 


Optionally,  specify  constraints  on  the  decision  variables 


o 


]]  Advanced  entry 


Constraints  i 


Type  Exclude 


optional  constraints  on  decision 
ariablesl 


Add  Constraint 


Add  Comment. 


Efficient  Frontier 


Delete 


Run 


Close 


Help 
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Control  Panel  -  OptQuest 


m 


Run  Analyze  Help 


Optimization  complete  -  all  decision  variables  have  been  fully 

enumerated. 


00:00 


o 


Time  (min:sec):  00:04 


10:00 


mmmmmimwmmiiliil 


Trials  10:QQ0 


10,000 


!► 
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Best  Solution:  Simulation  #  5 


Objectives 

Value 

Minimize  the  95%  Percentile  of  Tota  1C  riti  cal  Path  Days 

227.59 

Requirements 


Value 


Constraints 

Left  Side 

Right  Side 

—  Decision  Variables 

Value 

Arch  D  es  ig  nC  h  o  i  ce 

2.00 

CodeChoice 

2.00 

ReqtsChoice 

1.00 

UnitITChoice 

1.00 
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3 

OptOj  ic. 

**■  Results 

/£di 

l - 

t 

View 

Analyze^^references  Help 

✓  Best  Solution 


Solution  Analysis 


450.00 


i]j 


■«lj 

r_j 

Hi) 

Cl 

xP 

in 

ce 

Ci 

x= 

Cl 

ra 

u 


O 

"o 


Best  Solution  View 


Performance  Chart 


7  8  9 

Simulations 


Best  Solution: 


Sirr 


Software  Engineering  Institute  CarnedeMellon 


Robert  Stoddard  and  Dave  Zubrow 

©2010  Carnegie  Mellon  University 


SSTC  2010 


1 6  Total  Solutions  Solution  Analysis  View 


Objective 

—  Decision  Variables 

Rank 

Solution# 

Minimize  95%  Percentile 

T  atalCriticalPathDays 

Arc  hDesiqnCliaice 

CodeChoice 

ReqtsChaice 

UnitITChoice 

1 

5 

227.59 

2.00 

2.00 

1.00 

1.00 

2 

9 

258.40 

2.00 

1.00 

1.00 

1.00 

3 

10 

280.93 

2.00 

2.00 

1.00 

2.00 

4 

16 

288.51 

2.00 

2.00 

2.00 

1.00 

5 

15 

293.13 

1.00 

2.00 

1.00 

1.00 

6 

4 

311.57 

2.00 

1.00 

1.00 

2.00 

7 

7 

31  8.81 

2.00 

1.00 

2.00 

1.00 

8 

2 

323.84 

1.00 

1.00 

1.00 

1.00 

9 

1 

339.05 

2.00 

2.00 

2.00 

2.00 

10 

6 

344.50 

1.00 

2.00 

1.00 

2.00 

11 

3 

352.16 

1.00 

2.00 

2.00 

1.00 

12 

13 

369.57 

2.00 

1.00 

2.00 

2.00 

13 

12 

374.86 

1.00 

1.00 

1.00 

2.00 

14 

14 

382.32 

1.00 

1.00 

2.00 

1.00 

15 

11 

402.05 

1.00 

2.00 

2.00 

2.00 

16 

3 

432.65 

1.00 

1.00 

2.00 

2.00 
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PPM  Exercise  3: 
Predicting  Product 
Requirements  Change 
with  Linear  Regression 
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Statistical  Regression  Landscape 


SSTC  2010 


The  purpose  of  regression  is  to  perform  the  basic  task  of 
ANOVA  by  determining  whether  there  is  significant 
prediction  of  dependent  (y)  variable(s)  using  knowledge  of 
independent  (x)  variable(s). 


•  Example:  Can  the  defects  by  release  (y)  be  predicted  using 
knowledge  of  one  or  more  independent  variables  (x)s? 

•  Some  types  of  regression  (all  y’s  &  x’s  continuous  unless  noted  as 
discrete): 


Simple  linear 

1  y  &  1  “x” 

Multiple  linear 

1  “y”  &  multiple  “x”s 

Multivariate 

multiple  “y”s  &  1+  “x” 

Nonlinear 

nonlinear  version  of  the  above  types 

Logistic 

1  discrete  “y”  &  1+  “x”s 
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SSTC2010 

p  value  Summary 


Method 

Null 

Alternative 

P  <  0.05 

P  >  0.05 

Hypothesis 

Tests 

No  difference  exists; 
no  associations 

Two  items  are 
different;  association 
exists 

Accept 

alternative 

Accept 

null 

Tests  for 
Normality 

Data  follows  Normal 
Distribution 

Data  does  not  follow 
Normal  Distribution 

Accept 

alternative 

Accept 

null 

ANOVA 

No  difference  of  Y 
across  levels  of  x 

Difference  of  Y  exists 
between  1  +  levels  of 

X 

Accept 

alternative 

Accept 

null 

Regression 

x  factor  does  not  add 
value  to  model 

X  factor  adds  value 
to  model 

Accept 

alternative 

Accept 

null 

Chi-Square 

Two  discrete 
variables  are  not 
associated 

Two  discrete 
variables  are 
associated 

Accept 

alternative 

Accept 

null 

Logistic 

Regression 

x  factor  does  not  add 
value;  model  has  no 
significant  x’s 

X  factor  adds  value 
to  model;  model  has 
1+  significant  x’s 

Accept 

alternative 

Accept 

null 
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Slogan  to  Remember  p  Interpretation 


“When  the  p  is  low, 
the  null  must  go... 
When  the  p  is  high, 
the  null  must  fly” 
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Statistical  Regression  Analysis 

Y 


O 

0 

O 

t/> 


VO* 


Continuous 


Discrete 


AN  OVA 
and  Dummy 
Variable 
Regression 

Chi-Square, 
Logit  & 
Logistic 
Regression 

^tSorrelationX 
&  Linear 

^Regressions 

s,  Logistic 
/  Regression 
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Open  the  ReqtsChangeLinearRegression.jmp  file 

V _ _ _ ) 


4>\  ® 

©  \ 

ExpectedReqtChanges 

CustomerRelationshipAge 

ReadingLevel 

AgeOfReqt 

ReqtAnalystExperience 

Time  SinceCustStaffChange 

1 

0.56 

31.65 

7.29 

9.23 

48.55 

8.36 

2 

0.45 

26.34 

6.76 

8.77 

55.68 

7.4 

3 

0.76 

11.78 

8.68 

7.85 

46 

8.29 

4 

0.57 

44.99 

7.73 

9.21 

47.29 

7.94 

5 

0.9 

37.07 

9.39 

8.45 

45.4 

7.74 

6 

0.81 

22.81 

8.64 

8.38 

43.48 

7.54 

7 

0.65 

32.65 

8.8 

9.23 

46.12 

8.45 

8 

1.01 

2.79 

9.72 

10.04 

43.76 

6.75 

9 

0.88 

15.25 

9.47 

9.5 

46.32 

8.1 

10 

0.55 

31.22 

6.5 

8.77 

44.72 

7.43 

11 

0.85 

26.19 

8.77 

9.25 

57.85 

8.25 

12 

0.78 

18.77 

8.56 

10.2 

47 

8.54 

13 

0.79 

31.5 

8.05 

1 1 .29 

51.52 

6.88 

14 

1.09 

11.21 

9.58 

8.9 

47.65 

7.1 

15 

0.69 

48.57 

7.85 

9.56 

51.15 

7.91 

16 

1.13 

48.99 

11.45 

13.2 

44.78 

7.59 

17 

0.76 

28.75 

8.56 

7.98 

46.34 

7.2 

18 

0.86 

39.62 

8.92 

13.11 

43.24 

7.88 

19 

0.79 

22.55 

8.67 

11.76 

41.81 

7.09 

20 

0.66 

19.34 

7.94 

11.57 

53.48 

8.14 

21 

0.71 

48.31 

8.35 

9.65 

46.11 

8.85 

22 

0.77 

17.15 

8.75 

8.25 

53.25 

7.62 
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Factor 

Role 

Data  Type 

Description 

Expected  ReqtChanges 

Y  Outcome 

Continuous 

The  number  of  expected  changes 
that  will  occur  during  product 
development  with  a  given  product 
requirement 

CustomerRelationshipAge 

XI  Factor 

Continuous 

At  the  time  of  requirement 
formulation,  the  age  in  months  of 
the  relationship  with  the  customer 
of  the  product  development 

ReadingLevel 

X2  Factor 

Continuous 

The  reading  level  (grade  level) 
computed  for  the  requirement 
statement  (sentence  or  paragraph) 

AgeOfReqt 

X3  Factor 

Continuous 

The  age  in  months  of  the  product 
requirement  at  the  point  the 
requirement  is  identified  for  this 
product 

ReqtAnalystExperience 

X4  Factor 

Continuous 

The  experience  level  in  months  of 
the  Requirements  Analyst 

TimeSinceCustStaffChange 

X5  Factor 

Continuous 

At  the  time  of  requirement 
formulation,  the  number  of  months 
since  the  last  customer  staff  change 
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Model  Specification 


Select  Columns—i  —  Pick  Role  Variables 


^ExpectedReqtCh 

—  CustomerRelatio 

^ReadingLevel 

^AgeOfReqt 

^ReqtAnalystExpe 

^TimeSinceCustS 


^ExpectedReqtCI 

optional 


Weight 

optional  numeric 

Freq 

optional  numeric 

By 

optional 

Personality: 

Emphasis: 


Standard  Least  Squares 


Effect  Leverage 


Remove 


Construct  Model  Effects 


Add 


Cross 


Nest 


Macros 


Degree 


Attributes  ® 
Transform  T 

ED  No  Intercept 


CustomerRelationshipAge 

ReadingLevel 

AgeOfReqt 

R  e  qtAn  a  ly  stExp  e  ri  e  n  c  e 
TimeSinceCustStaffChange 


J 
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w  Summary  of  Fit 

jmzi3 

^RSquare  Adj 

0. 90249 

Mean  of  Response 
Observations  (or  Sum  Wgts) 

0.844075 

400 

w  Analysis  of  Variance 

1 

Sum  of 

Source  DF  Squares  Mean  Square  F  Ratio 

Model  5  14.825457  2.96509  739.5833 

Error  394  1.579600  0.00401  Prob  >  F 

C.  Total  399  16.405058  <.0001* 

Parameter  Estimates 

Term 

Estimate 

Std  Error  t  Ratio  a 

^rob>|tf'\ 

Intercept 

0.0811043 

0.064497  1.2/ 

0.2093  \ 

CustomerRelationshipAge 

-0.000144 

0.000262  -035 

0.5833  \ 

ReadingLevel 

0.1574728 

0.002647  59.49 

<  0001*  ] 

AgeOfReqt 

0.0004456 

0.001195  0.17 

0.7094  / 

R  e  qtAn  a  lystExp  e  ri  e  n  c  e 

-0.003763 

0.000862  -4.3\ 

<.0001*  J 

TimeSinceCustStaffChange 

-0.058817 

0.005759  -10.2 1  \ 

,<  0001*/ 
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r  ^ 

:  Fit  Model 

-Imx 

\w  T  Model  Specification 

. 

Select  Columns— i  r  Pick  Role  Variables 


^ExpectedReqtCh 

^CustomerRelatio 

^ReadingLevel 

^AgeOfReqt 

^ReqtAnalystExpe 

^TimeSinceCustS 


Personality:  Standard  Least  Squares 


^ExpectedReqtCI 

optional 


Emphasis:  Effect  Leverage 


Weight 

optional  numeric 

Freq 

optiona!  numeric 

By 

optional  G 

Help 


Run  Model 


Construct  Model  Effects 


Add 

CustomerRelationshipAge 

Cross 

AgeOfReqt  ) 

Nest 

TimeSinceCustStaffChange 

Macros  v 

Degree  2 

Attributes  ® 

Transform  T 

CD  No  Intercept 

J 
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*  Model  Specification 


Select  Columns— i  r  Pick  Role  Variables 


^ExpectedReqtCh 

^CustomerRelatio 

^ReadingLevel 

^AgeOfReqt 

A  R  e  qtAn  a  ly  stExp  e 

^TimeSinceCustS 


Y 

A  ExpectedReqtCI 

optional 

Weight 

optional  numeric 

Freq 

optional  numeric 

By 

optional 

Personality:  Standard  Least  Squares  v 
Emphasis: 


Effect  Leverage 


Remove 


Construct  Model  Effects 


Add 

CustomerRelationshipAge 

ReadingLevel 

R  e  qtAn  a  ly  stExp  e  ri  e  n  c  e 
TimeSinceCustStaffChange 

Cross 

Nest 

Macros  v 

Degree  2 

Attributes  T 

Transform  T 

CH  No  Intercept 

j 


Software  Engineering  Institute  GarnedeMellon 


Robert  Stoddard  and  Dave  Zubrow 

©2010  Carnegie  Mellon  University 


SSTC  2010 


Summary  of  Fit 


RSaiiac&i 


^Square  Adj 

Root  Meai  i  SL|Udl  u  El  I  ui 
Mean  of  Response 
Observations  (or  Sum  Wgts) 


0.902703 

TJ.Ubj^W 


0.844075 

400 


Analysis  of  Variance 


Source 

Model 


Sum  of 

DF  Squares  Mean  Square 

4  14.824900  3.70623 


F  Ratio 

926.4638 


Error  395  1.580158 

C.  Total  399  16.405058 

0.00400  Prob  >  F 

<  0001* 

Parameter  Estimates 

Term 

Estimate 

Std  Error 

t  Ratio/ 

/Prob>|t| 

Intercept 

0.0835877 

0.064082 

1.3/ 

0.1929 

CustomerRelationshipAge 

-0.000137 

0.000261 

-0.33 

0.5998 

ReadingLevel 

0.1575641 

0.002633 

59.35 

<ooor 

ReqtAnalystExperience 

-0.003748 

0.00086 

4  31S 

<ooor 

TimeSinceCustStaffChange 

-0.058785 

0.005752 

-10.22\ 

v  <ooor 
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T  Model  Specification 


Select  Columns— i  ffi  Pick  Role  Variables 


A  Expected  Re  qtCh 

^CustomerRelatio 

^ReadingLevel 

^AgeOfReqt 

^ReqtAnalystExpe 

^TimeSinceCustS 


Y 

A  ExpectedReqtCI 

optional 

Weight 

optional  numeric 

Freq 

optional  numeric 

By 

optional  t 

Personality:  Standard  Least  Squares 


Emphasis:  Effect  Leverage 


Construct  Model 


Add 


Cross 


Nest 


Macros 


Degree 

Attributes  ® 
Transform  T 

EH  No  Intercept 


CustomerRelationshipAge 


jeftrigLLHil  — 

R  e  qtAn  a  ly  stExp  e  ri  e  n  c  e 
TimeSinceCustStaffChange 


J 
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Model  Specification 


Select  Columns 


A  Exp  e  cte  d  ReqtC  h 
^CustomerRelatio 
iReadingLevel 
^AgeOfReqt 
^ReqtAnalystExpe 
JTimeSinceCustS 


Weight 

optional  numeric 

Freq 

optional  numeric 

By 

optional 

Pick  Role  Variables 

Personality: 

Y 

A  ExpectedReqtCI 

Emphasis: 

Standard  Least  Squares 


Effect  Leverage 


Remove 


Construct  Model  Effects 


Add 

ReadingLevel 

R  e  qtAn  a  ly  stExp  e  r  i  e  n  c  e 
TimeSinceCustStaffChange 

Cross 

Nest 

Macros  v 

Degree  2 

Attributes  © 

Transform  T 

ED  No  Intercept 

j 
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Summary  of  Fit 


^Square  Adj 
Root  Mel 
Mean  of  Response 
Observations  (or  Sum  Wgts) 


0.844075 

400 


Analysis  of  Variance 


Source 

Model 
Error 
C.  Total 


Sum  of 

DF  Squares  Mean  Square 

3  14.823797  4.94127 

396  1.581261  0.00399 

399  16.405058 


F  Ratio 

1237.457 

Prob  >  F 

<.0001* 


Parameter  Estimates 


Term 

Intercept 

ReadingLevel 

ReqtAnalystExperience 

TimeSinceCustStaffChange 


Estimate 

Std  Error 

t  Ratio/ 

Prob>|t| 

0.079986 

0.063656 

1.2* 

0.2097 

0.1576274 

0.002628 

59.96 

<.0001* 

-0.003749 

0.000859 

-4.3ft 

<.0001* 

-0.058869 

0.005745 

-10.25\ 

,  <.0001* 
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Residual  by  Predicted  Plot 


0.3  0.5  0.7  0.9  1.1  1.3  1.5  1.7 


ExpectedReqtChanges 

Predicted 
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sf  JMP  -  [ReqtsChangeLinearRegression-  Fit  Least  Squares] 

SI  File  Edit  Tables  Rows  Cols  DOE  Analyze  Graph  Tools 

View  Window  Help 

liff  isl  fl  ^  l&lHl  Kfr|  ?  +  O  |l  j  |  ReqtsCha 


— “""““‘A?dReqtChanges 

egression  Reports  ►  l  ^ 


Estimates 


Effect  Screening 
Factor  Profiling 
Row  Diagnostics 
Save  Columns 
Script 


Show  Prediction  Expression 


Sorted  Estimates  ^ 

Expanded  Estimates 
Sequential  Tests 
Custom  Test- 

Joint  Factor  Tests 

Inverse  Prediction... 

Parameter  Power 
Correlation  of  Estimates 


1  I  I  I  I  I  I  I  I  I  I  I  I  I 

0.3  0.5  0.7  0.9  1.1  1.3  1.5  1.7 


ExpectedReqtChanges 

Predicted 

P<  0001  RSq=0.90 
RMS  E= 0.0632 


_ 


Display  the  text  of  the  prediction 
formula  with  the  estimates  inserte 


_ 
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w  Prediction  Expression 

0 . 0799860 1 6248 

+  0.1 5762743466752  *  ReadingLevel 
+  -0. 0037486 144177 
*  R  eqtAnalystE  xperi  ence 
+  -0. 058869 1 875889 
*TimeSinceOustStaff  Change 
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SSTC  2010 


=f  JMP  -  [ReqtsChangeLfnearRegressfon-  Fit  Least  Squares] 


r  File  Edit  Tables  Rows  Cols  DOE  Analyze  Graph  Tools  View  Window  Help 


1  ^  id  A  % 

life] 

?*®<lWpQi+l 

ID  5  C>  O 

ReqtsChangeL 


ression  Reports 
Estimates 
Effect  Screening 
Factor  Profiting 
Row  Diagnostics 


Save  Columns 


^*3 

:r 


BdReqtChanges 


ed  Plot 


Script 


0.3  0.5  0.7 

ExpectedR 
Predicted 
P<.0001  R! 
RMSE=0.ffl 


Summary  of  Fit 

RSquare 
RSquare  Adj 
Root  Mean  Square  Error 


Prediction  Formula 
Predicted  Values 
Residuals 


Mean  Confidence  Interval 


Indiv  Confidence  Interval 
Studentized  Residuals 
Hats 

Std  Error  of  Predicted 
Std  Error  of  Residual 
Std  Error  of  Individual 
Effect  Leverage  Pairs 
Cook's  D  Influence 
Std  Err  Pred  Formula 
Save  Coding  Table 


You  will  need  to  do  this 
menu  3  times  to  make  each 
of  the  three  choices: 

Predicted  Values 

Mean  Confidence  Intervals 

Jndiv  Confidence  Intervals 


The  confider 
expected  valt 
realization).  This  encompasses  the 
variation  in  the  estimation,  but  not  in 
the  response. 


u.uudiyi 


jC 


6 
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i  x  © 

©  \ 

Predicted 

ExpectedReqtChan 

Lower  95%  Mean 
ExpectedReqtChanges 

Upper  95%  Mean 
ExpectedReqtChanges 

Lower  95%  Indiv 
ExpectedReqtChanges 

Upper  95%  Indiv 
ExpectedReqtChanges 

1 

0.55494838 

0.54354212 

0.56635463 

0.43019439 

0.67970237 

2 

0.50119264 

0.48071519 

0.52167008 

0.37528481 

0.62710046 

3 

0.78773032 

0.77996762 

0.79549302 

0.66325657 

0.91220407 

4 

0.65375276 

0.64466949 

0.66283604 

0.52918968 

0.77831584 

5 

0.93427302 

0.92589222 

0.94265383 

0.80975919 

1.05878685 

6 

0.83502362 

0.82398931 

0.84605794 

0.71030309 

0.95974416 

7 

0.79677671 

0.78830532 

0.8052481 

0.67225675 

0.92129667 

3 

1.0507183 

1.0333979 

1.06803869 

0.92528524 

1.17615135 

9 

0.92224158 

0.91497575 

0.92950742 

0.79779783 

1.04668534 

10 

0.49952824 

0.48310202 

0.51595446 

0.37421553 

0.62484096 

11 

0.75985048 

0.74183342 

0.77786753 

0.63431933 

0.88538163 

12 

0.75034912 

0.7414727 

0.75922554 

0.62580095 

0.87489729 

13 

0.75073824 

0.7338668 

0.76760968 

0.62536639 

0.87611009 

14 

0.99346413 

0.98080803 

1.00612023 

0.86858966 

1.1183386 

15 

0.65996448 

0.64952963 

0.67039932 

0.53529555 

0.78463341 

16 

1.27014006 

1.25421652 

1.28606359 

1.14489224 

1.39538787 

17 

0.83170791 

0.82004771 

0.84336812 

0.70693045 

0.95648538 

18 

0.86004345 

0.84999154 

0.87009535 

0.73540599 

0.98468091 

19 

0.87250377 

0.85688522 

0.88812231 

0.74729436 

0.99771317 

20 

0.65187676 

0.6392197 

0.66453382 

0.5270022 

0.77675133 

21 

0.70233417 

0.69025672 

0.71441163 

0.57751703 

0.82715132 

22 

0.81 102914 

0.7987054 

0.82335289 

0.68618792 

0.93587036 
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Predicting  Delivered 
Defects  with  Dummy 
Variable  Regression 
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Dummy  Variable  Regression 


SSTC  2010 


The  purpose  of  Dummy  Variable  Regression  is  to  predict  a 
continuous  Y  outcome  using  a  combination  of  continuous 
and  discrete  x  factors. 
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Statistical  Regression  Analysis 

Y 


O 

0 

O 

t/> 


c 
o 
VO 


Discrete 


anovaX 

and  Dummy  1 
Variable 
^  Regression y 

Chi-Square, 
Logit  & 
Logistic 
Regression 

CDfFelatron 
&  Linear 
Regression 

Logistic 

Regression 
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SSTC  2010 


Open  the  DeliveredDefectsDummyVariableRegression.jmp  file 


DeliveredDefects 

InspectionDefects 

InspectionCoverage 

InspectionTeamExperience 

PercentNewCode 

InspectionType 

UnitTestType 

0.52 

7.21 

81.46 

52.55 

75.18 

0 

0 

0.56 

6.95 

81.77 

46.76 

76.68 

oC 

F  0 

0.58 

9.69 

81.94 

48.12 

77.43 

0 

0 

0.32 

727 

90.75 

36.93 

75.45 

1 

0 

0.6 

6.1 

74.11 

36.5 

77.59 

0 

0 

0.54 

6.53 

76.42 

45.37 

78 

0 

0 

0.41 

5.72 

79.52 

31.42 

78.35 

0 

1 

0.45 

6.94 

88.68 

48.64 

81.13 

0 

0 

0.59 

6.54 

70.69 

27.42 

76.8 

0 

0 

0.6 

6.3 

78.14 

22.48 

75.93 

0 

0 

0.55 

6.52 

76.01 

33.9 

82.39 

0 

0 

0.6 

9.67 

77 

49.69 

80.04 

0 

0 

0.66 

6.54 

65.28 

45.86 

76.84 

1 

0 

0.6 

8.45 

79.48 

40.3 

80.81 

0 

0 

0.53 

5.77 

77.15 

38.23 

78.51 

0 

0 

0.52 

5.77 

78.98 

39.47 

79.84 

0 

0 

0.55 

5.68 

73.57 

29.03 

80.74 

0 

0 

0.41 

8.57 

80.08 

24.82 

76.55 

0 

1 

0.45 

8.99 

85.46 

34.95 

80.09 

0 

1 

0.55 

6.38 

76.66 

27.05 

82.46 

0 

0 

0.43 

5.83 

80.14 

33.77 

78.74 

1 

1 
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Factor 

Role 

Data  Type 

Description 

DeliveredDefects 

Y  Outcome 

Continuous 

Delivered  Defect  Density  normalized  to 
KSLOC  for  a  given  feature 

Inspection  Defects 

XI  Factor 

Continuous 

Inspection  Defect  Density  normalized 
to  KSLOC  for  a  given  feature 

InspectionCoverage 

X2  Factor 

Continuous 

The  percentage  of  inspection  criteria 
implemented  across  the  code  files  for 
a  given  feature 

InspectionTeamExp 

erience 

X3  Factor 

Continuous 

The  average  domain  experience  in 
months  of  the  participants  of  the  peer 
review  of  the  feature 

PercentNewCode 

X4  Factor 

Continuous 

The  percent  of  new  code  within  the 
feature 

InspectionType 

X5  Factor 

Nominal 

A  factor  which  reflects  whether  an 
informal  peer  review  (0)  vs  a  formal 
inspection  (1)  occurred  for  the  feature 

UnitTestType 

X6  Factor 

Nominal 

A  factor  which  reflects  whether 
informal  unit  testing  (0)  vs  formal  unit 
testing  (1)  occurred  for  the  feature 
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JMP  -  Deli  vere  d  DefectsDu  m  my Va  ri  a  b  leRegre  ssi  o  n 

Analyze 


File  Edit  Tables  Rows  Cols  DOE 

Distribution 
lls  Fit  Y  by  X 
^  Matched  Pairs 


Graph  Tools  View  Window  H 


Fit  Model 


Modeling 

Multivariate  Methods 
Reliably  and  Survival 


6 


8 


p 


► 

► 

► 


acts 


0.52 


0.56 


0.58 


0.32 


0.6 


0.54 


0.41 


0.45 


InspectionDefects 


7.21 


6.95 


9.69 


7.27 


6.1 


6.53 


5.72 


6.94 


0.59 
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Summary  of  Fit 


<£ 


^jq1 


*64 


RSquare  Adj 

Mean  of  Response 
Observations  (or  Sum  Wgts) 


0.844753 

u.uJJtlljS 


0.183701 

589 


Analysis  of  Variance 


Source 

Model 

Error 


Sum  of 

DF  Squares  Mean  Square  F  Ratio 

6  3.6198962  0.603316  534.2531 

582  0.6572352  0.001129  Prob  >  F 


C.  Total  588  4.2771314 

<.0001* 

Parameter  Estimates 

Term 

Estimate 

Std  Error 

t  Ratio  J 

^rob>|t| 

Intercept 

1.0617488 

0.061206 

17.3 / 

<.0001* 

InspectionDefects 

0.0098332 

0.001301 

7M 

<.0001* 

InspectionCoverage 

-0.009272 

0.000278 

-33. B9 

<.0001* 

InspectionTeamExperience 

-0.000149 

0.000132 

-1.13 

0.2588 

PercentNewCode 

0.0009154 

0.000705 

ih 

0.1947 

lnspectionType[0] 

0.023847 

0.001518 

15.7\ 

<.0001* 

UnitTestType[0] 

0.0609747 

0.00147 

41.47 V 

s<ooor 
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■ 


Fit  Model 


Ba 


T  Model  Specification 


Select  Columns—i  — Pick  Role  Variables 


^DeliveredDefects 

-^InspectionDefect 

^InspectionCoverz 

^InspectionTeamE 

^PercentNewCodi 

il«lnspectionType 

il«UnitTestType 


Y 

A  DeliveredDefect 

optional 

Weight 

optional  numeric 

Freq 

|  optional  numeric 

By 

optional  G 

Personality:  Standard  Least  Squares  v 
Emphasis: 


Effect  Leverage 


Construct  Model  Effects 


Add 


InspectionDefects 

HiuilCUVUI  jyu 


Cross 


Nest 


InspectionTeamExperience 


ffttlniftrCnde 


Macros 


InspectionType 

UnitTestType 


Degree 

Attributes  T 
Transform  s 

EH  No  Intercept 


j 
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Model  Specification 


Select  Columns— i  —  Pick  Role  Variables 


^DeliveredDefects 
^InspectionDefect 
IlnspectionCovers 
^InspectionTeamE 
PercentNewCodi 
AlnspectionType 
AlJnitTestType 


Y 

A  DeliveredDefect 

optional 

Weight 

optional  numeric 

Freq 

optional  numeric 

By 

optional 

Personality:  Standard  Least  Squares  v 
Emphasis 


Effect  Leverage 


Help 


Recall 


Remove 


Construct  Model  Effects 


Add 

InspectionDefects 

InspectionCoverage 

PercentNewCode 

InspectionType 

UnitTestTvpe 

Cross 

Nest 

Macros  v 

Degree  2 

Attributes  T 

Transform  T 

ED  No  Intercept 

j 
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Summary  of  Fit 


finiinrr- 


Square  Adj 
Root  Mean  SyUclta  EHUI 
Mean  uf  Respunse 
Observations  (or  Sum  Wgts) 


0.844671 
tn)S3513 


0.483701 

589 


Analysis  of  Variance 


Source 

Model 
Error 
C.  Total 


Sum  of 

DF  Squares  Mean  Square  F  Ratio 

5  3.6184537  0.723691  640.5434 

583  0.6586777  0.001130  Prob  >  F 

588  4.2771314  <.0001* 


Parameter  Estimates 


Term 

Estimate 

Std  Error 

t  Ratio  y 

Trob>|t| 

Intercept 

1.0562782 

0.061028 

17.3  / 

<.0001* 

InspectionDefects 

0.009816 

0.001302 

7M 

<.0001* 

InspectionCoverage 

-0.009266 

0.000278 

-33.37 

<.0001* 

PercentNewCode 

0.0009039 

0.000705 

1  2E 

0.2004 

lnspectionType[0] 

0.0239018 

0.001518 

1 5.73L 

<.0001* 

UnitTestTvpe[01 

0.0609821 

0.00147 

41.47> 

<<.0001* 
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Select  Columns— i  — Pick  Role  Variables 


^DeliveredDefects 

^InspectionDefect 

^Inspection  Cover; 

^InspectionTeamE 

^PercentNewCodi 

il«  InspectionType 

illnitTestType 


Y 

J  DeliveredDefect 

optional 

Weight 

optional  numeric 

Freq 

optional  numeric 

By 

V. 

Personality: 

Emphasis: 


Standard  Least  Squares 


Effect  Leverage 


Construct  Model  Effects 


Add 


Cross 


Nest 


Macros 


Degree 


Attributes  T 
Transform  2 

I~1  No  Intercept 


InspectionDefects 
ffnu  overage" 


PercentNewCode 


InspLullUl  1 1  ,/pi 
UnitTestType 


Run  Model 


j 
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Model  Specification 


Select  Columns— i  —  Pick  Role  Variables 


^DeliveredDefects 

^InspectionDefect 

^InspectionCoverz 

^InspectionTeamE 

^PercentNewCodi 

il«lnspectionType 

il«UnitTestType 


Y 

A  DeliveredDefect 

optional 

Weight 

optional  numeric 

Freq 

optional  numeric 

By 

optional 

Personality: 

Emphasis: 


Standard  Least  Squares 


Effect  Leverage 


Remove 


Construct  Model  Effects 


Add 

InspectionDefects 

InspectionC  overage 

InspectionType 

UnitTestType 

Cross 

Nest 

Macros  v  | 

Degree  2 

Attributes  T 

Transform  2 

EH  No  Intercept 

j 
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Square  Adj 
Root  Mean  square 
Mean  of  Response 
Observations  (or  Sum  Wgts) 


0.844508. 

33631 
0.483701 
589 


Analysis  of  Variance 


Source 

Model 


DF 

4 


Sum  of 

Squares  Mean  Square 


3.6165970 


0.904149 


F  Ratio 

799.3879 


Error  584 

C.  Total  588 

0.6605344 

4.2771314 

0.001131 

Prob  >  F 

<  0001* 

Parameter  Estimates 

Term 

Estimate 

Std  Error 

t  Ratio/ 

/Prob>|t| 

Intercept 

1.1279627 

0.024456 

46.  l/ 

<.0001* 

InspectionDefects 

0.0097196 

0.0013 

7.48 

<.0001* 

InspectionCoverage 

-0.00927 

0.000278 

-33.3c 

<000  r 

lnspectionType[0] 

0.023858 

0.001518 

15.7V 

<.0001* 

UnitTestType[0] 

0.0608499 

0.001468 

41.46> 

s<  ooor 
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JMP  -  [DelfveredDefectsDummyVanableRegression-  Fit  Least  Squares] 

;■  Rle  Edit  Tables  Rows  Cols  DOE  Analyze  Graph  Tools  View  Window  Help 

Deli1 vered  DefectsD  u  it 

Windows 


-  ^  Delivered  Defects 
*  DeliveredDefec 
■  *  Fit  Model 


Regression  Reports 


Estimates 


Effect  Screening 
Factor  Profiling 
Row  Diagnostics 
Save  Columns 
Script 


B 


ed  Defects 


Show  Prediction  Expression 


Sorted  Estimates 
Expanded  Estimates 
Indicator  Parameterization  Estimates 
Sequential  Tests 
Custom  Test... 

Joint  Factor  Tests 
Inverse  Prediction... 

Parameter  Power 
Correlation  of  Estimates 


^  - 

Display  th 
formula  w 


f - - to - urr 

DeliveredDefects  Predicted 
P<  0001  R3q=0.85 


-i —i 

u 

ed 

LI¬ 

ED 

Q 

"O 

ed 

i_ 

ED 

> 

13 
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JMP  -  [DeliveredDefectsDummyVariabLeRegression-  Fit  Least  Squares] 


i-  File  Edit  Tables  Rows  Cols  DOE  Analyze  Graph  Tools  View  Window  Help 


SSTC  2010 


j  j  ii  a  id  ^ 


Windows 


-  Delivered  DefectsC 

Delivered  Defec 
Fit  Model 


i|||[ 


Delivered  De 


'egression  Reports 
Estimates 
Effect  Screening 
Factor  Profiling 
Row  Diagnostics 


Save  Columns 


'edDefects 


You  will  need  to  do  this 
menu  3  times  to  make  each 
of  the  three  choices: 

Predicted  Values 

Mean  Confidence  Intervals 

Jndiv  Confidence  Intervals 


Root  Mean  Square  Ern 


ed  Plot 


Prediction  Formula 
Predicted  Values 
Residuals 


Mean  Confidence  Interval 


Indiv  Confidence  Interval 
Studentized  Residuals 
Hats 

Std  Error  of  Predicted 
Std  Error  of  Residual 
Std  Error  of  Individual 
Effect  Leverage  Pairs 
Cook's  D  Influence 
Std  Err  Pred  Formula 
Save  Coding  Table 


* 


The  confidei 
expected  v; 
realization). ' 
variation  in  t 
the  respons 
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Predicted 

DeliveredDefect 

Lower  95%  Mean 

DeliveredDefects 

Upper  95%  Mean 
DeliveredDefects 

Lower  95%  Indiv 

DeliveredDefects 

Upper  95%  Indiv 
DeliveredDefects 

0.52762634 

0.52375953 

0.53149316 

0.46146059 

0.5937921 

0.52222559 

0.5183626 

0.52608858 

0.45606005 

0.58839112 

0.54728146 

0.53935391 

0.55520901 

0.48075477 

0.61380816 

0.39437653 

0.38606592 

0.40268715 

0.3278031 

0.46094996 

0.58497103 

0.57942818 

0.59051389 

0.51868621 

0.65125586 

0.56773709 

0.5632922 

0.57218199 

0.50153504 

0.63393915 

0.40942786 

0.40306404 

0.41579169 

0.34306934 

0.47578639 

0.45807366 

0.45207219 

0.46407514 

0.391 74891 

0.52439842 

0.62095058 

0.61444472 

0.62745645 

0.55457829 

0.68732288 

0.54955743 

0.54524504 

0.55386981 

0.48336413 

0.61575072 

0.57144054 

0.56688035 

0.57600073 

0.50523064 

0.63765044 

0.59288017 

0.584991 1 5 

0.6007692 

0.52635806 

0.65940229 

0.62338452 

0.6138448 

0.63292425 

0.55664652 

0.69012253 

0.55803299 

0.55276691 

0.56329907 

0.49177073 

0.62429525 

0.55358319 

0.54838539 

0.55878099 

0.48732632 

0.61984005 

0.53661935 

0.53166763 

0.54157106 

0.47038133 

0.60285736 

0.58589452 

0.57962983 

0.59215921 

0.51954543 

0.65224361 

0.43193765 

0.42581413 

0.438061 18 

0.36560174 

0.49827356 

0.38614805 

0.37870953 

0.39358656 

0.31967785 

0.45261824 

0.56405439 

0.5595407 

0.56856808 

0.49784768 

0.6302611 

0.35703371 

0.35012345 

0.36394397 

0.29062056 

0.42344686 

0.43740385 

0.43126051 

0.4435472 

0.37106611 

0.50374159 

n  ^Q9n7cm 

n 

n 
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Statistical  Regression  Analysis 

Y 


© 

CD 

Continuous 

Discrete 

AN  OVA 

CKi-Squar^v 

Vlr 

O 

(/) 

and  Dummy 

/  Logit  &  \ 

M 

■ 

Q 

1 

Variable 

/  Logistic 

1  S 

Regression 

Regression 

o 

3 

C 

■  HHI 

c 

o 

VO 

Correlation 
&  Linear 
Regression 

\  Logistic 
'Regression/ 
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The  purpose  of  logistic  regression  is  to  predict  a  discrete 
(attribute)  Y  outcome  using  continuous  X  factors. 

Logistic  regression  belongs  to  the  class  of  models  generally 
referred  to  as  log-linear  models. 


Types  of  logistic  regression  analysis  include  the  following: 

•  nominal  -  a  nominal  Y  is  predicted  (e.g.,  categorical  without 
ordering) 

•  ordinal  -  an  ordinal  Y  is  predicted  (e.g.,  categorical  with  ordering) 

•  binary  -  a  binomial  Y  is  predicted  (e.g.,  Y  is  categorical  with  only 
two  possible  values) 
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CustomerSatisfactionExerciseWithOrdinalLogisticRegression.jmp  file 


PredictedCustomerSat 

AvgAgeUnresolvedCust 

QuestionsAtCoding 

AvgAgeUnresolvedDev 

QuestionsAtCoding 

AvgWeeklylnPerson 

Meetings 

AvgWeekly 

Telecons 

1 

20.27 

49.02 

0.15 

4.91 

3 

19.86 

47.53 

0.99 

4.15 

4 

19.52 

51.5 

3.57 

3.14 

2 

18.56 

49.02 

0.59 

4.58 

3 

20.45 

46.22 

1.34 

2.48 

3 

20.4 

48.22 

2.49 

0.84 

3 

19.42 

45.43 

0.41 

4.02 

4 

19.85 

48.03 

1.81 

4.69 

4 

19.98 

47.25 

3.13 

3.95 

2 

19.99 

48.93 

1.39 

1  67 

1  3 

19.11 

47.92 

1.52 

1.76 

3 

20.95 

47.93 

0.69 

5.17 

3 

19.5 

41.1 

2.8 

5.31 

3 

19.72 

42.94 

1.44 

4.18 

3 

20.69 

44.88 

1.27 

2.53 

2 

18.95 

50.49 

0.31 

6.12 
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ReqtsElicitation 

Method 

UnresolvedCustQuestions 

AtCoding 

UnresolvedDevQuestions 

AtCoding 

1.00 

28.8 

36.33 

2.00 

30.88 

41.15 

1.00 

23.74 

47.75 

1.00 

26.16 

49.61 

2.00 

32.35 

43.4 

1.00 

27.56 

42.92 

3.00 

25.53 

45.85 

2.00 

26.22 

47.87 

1.00 

31.26 

42.48 

1.00 

31.62 

41.47 

1.00 

26.34 

46.71 

2.00 

31.38 

51.41 

1.00 

24.47 

45.12 

1.00 

30.6 

44.8 

2.00 

27.59 

47.24 

1.00 

27.6 

48.9 

1.00 

29.18 

42.58 
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PredictedCustomerSat 

AvgAgeUnresolvedCus 

tQuestionsAtCoding 

Avg  AgeLI  n  resolved  Dev 
QuestionsAtCoding 

AvgWeeklylnPersonM 

eetings 

AvgWeeklyTelecons 


ReqtsElicitationMethod 

UnResolvedCustQuest 

ionsAtCoding 

UnResolvedDevQuesti 

onsAtCoding 


Role 

Data  Type 

Description 

Y  Outcome 

Ordinal 

Very  Low=1;  Low=2;  Medium=3;  High=4; 

Very  High=5 

XI  Factor 

Continuous 

Average  Age  in  Work  Days  of  Unresolved 
Questions  From  Customer  at  the  Beginning 
of  Coding  Phase 

X2  Factor 

Continuous 

Average  Age  in  Work  Days  of  Unresolved 
Questions  From  Developer  Team  at  the 
Beginning  of  Coding  Phase 

X3  Factor 

Continuous 

Average  Number  of  Face  to  Face  meetings 
per  week  between  the  Development  Team 
and  the  Customer 

X4  Factor 

Continuous 

Average  Number  of  Teleconference  Calls 
held  each  Week  between  the  Development 
Team  and  the  Customer 

X5  Factor 

Nominal 

Strictly  Spec  Driven=1;  lnterview=2; 
Prototyping=3 

X6  Factor 

Continuous 

Number  of  Unresolved  Questions  From 
Customer  at  the  Beginning  of  Coding  Phase 

X7  Factor 

Continuous 

Number  of  Unresolved  Questions  From 
Developer  Team  at  the  Beginning  of  Coding 
Phase 
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|Q  File  Edit  Tables  Rows  Cols  DOE 
1]  13  Bi  0  d  d3  Distribution 


Graph  Tools  View  Window  Help 


Windows 


iQ  Custom  erSattsf  Matched  Pairs 


I!*  Fit  Y  by  X 


Fit  Model 


% 


Modeling 

Multivariate  Methods 
Reliability  and  Survival 


©Columns  (8/1 ) 


PredictedCustomerSa 


tf  p  q*  +  el]™c>o 

j 

j 

j 

j 

j 

j 

ustomed 

\© 

mp  \ 

PredictedCustomerSat 

AvgAg 

Que 

1 

3 

3 

4 

2 

3 

6 

3 

7 

3 

8 

4 

9 

4 

10 

2 

3 

12 

3 

13 

3 

1 A 

T! 
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Model  Specification 


Select  Columns— i  r  Pick  Role  Variables 


PredictedCuston 
^AvgAgeUnresolvt 
^AvgAgeUnresolvt 
^AvgWeeklylnPer; 
^AvgWeeklyTelec 
■■■  ReqtsElicitationrv 
^UnresolvedCustC 
^UnresolvedDevG 


JPredictedCustor 

optional 


Weight  optional  numeric 


Freq  optional  numeric 


By  optional 


Personality:  Ordinal  Logistic 


Remove 


Add 

AvgAgeUnresolvedCustQuestionsAtC 

AvgAgellnresolvedDevQuestionsAtCc 

AvgWeeklylnPersonMeetings 

AvgWeeklyT  elec  ons 

ReqtsElicitationMethod 

UnresolvedCustQuestionsAtCoding 

UnresolvedDevQuestionsAtCoding 

Cross 

Nest 

Macros  v 

Degree  2 

Attributes  T 

Transform  T 

ED  No  Intercept 

j 
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Ordinal  Logistic  Fit  for  PredictedCustomerSat 

► 

Iteration  History 

▼ 

Whole  Model  Test  

Model 

Difference 

Full 

Reducei 


-LogLikelihood 

100.56109 

8.84501 
inn  anfiin 


DF 

8 


ChiSquare 

201.1222 


Prob>ChiSq 

<.0001* 


JSquare  (U) 

ODbefUdLiun^  (ui  GumWgts^" 
Converged  by  Gradient 


0.919 

"So 


Lack  Of  Fit 


Source 

Lack  Of  Fit 

Saturated 

Fitted 


DF  -LogLikelihood 

348  8.8450140 

356  0.0000000 

8  8.8450140 


ChiSquare 

17.69003 

Prob>ChiSq 

1.0000 
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Parameter  Estimates 


Term 

Intercept^] 

Intercept^] 

Intercept^] 

Intercept^] 

AvgAgeUnresolvedCustQuestionsAtCoding 
AvgAgeUnresolvedDevQuestionsAtCoding 
AvgWeeklylnPersonMeetings 
AvgWeeklyT  elecons 
ReqtsElicitationMethod[1 . 00] 
ReqtsElicitationMethod[2. 00] 
UnresolvedCustQuestionsAtCoding 
UnresolvedDevQuestionsAtCoding 


Estimate 

92.9334278 

148.597425 

203.99553 

254.214635 

1.96123574 

-0.4579541 

-35.583689 

-0.7033199 

31.0735426 

-4.398162 

-1.7209171 

=2.152774 


Std  Error  ChiSquare 


51.810259 

71.058627 

92.369135 

112.83919 

1.0437171 

0.5633572 

15.138283 

0.8018375 

13.061873 

2.3425233 

0.7953154 

0.8611309 


Software  Engineering  Institute  GarnedeMellon 


Robert  Stoddard  and  Dave  Zubrow 

©2010  Carnegie  Mellon  University 


I 


Fit  Model 


m 


▼ 


Model  Specification 


Select  Columns  Pick  Role  Variables 


PredictedCuston 
^AvgAgeUnresolvt 
^AvgAgeUnresolvE 
^AvgWeeklylnPers 
JAvgWeeklyTelec 
ilLReqts  Elicitation^ 
^UnresolvedCustC 
^UnresolvedDevG 


Y 

JPredictedCustor 

optional 

Weight 

optional  numeric 

Freq 

\optionafnum  eric  ( 

By 

optional 

Personality:  Ordinal  Logistic 


Help 


Run  Model 


Construct  Model  Effects 


Add 


Cross 


Nest 


Macros 


Degree 

Attributes  © 
Transform  © 

CH  No  Intercept 


Avg  Ag  eUnresolvedDevQuestion  s  AtC 


AvyWeeklVIl  IHtJI  LUI IMUULII  IIJs 
AvgWeeklyT  elecons 
R  e  qts  E I  i  c  itati  o  n  M  eth  o  d 
UnresolvedCustQuestionsAtCoding 
UnresolvedDevQuestionsAtCoding 
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Select  Columns— i  —  Pick  Role  Variables 


PredictedCuston 
^AvgAgeUnresolvE 
^AvgAgeUnresolvE 
^Avg  Weekly  InPen 
^AvgWeeklyTelec 
■I-  R  e  qt  s  E I  i  c  itati  o  n  N' 
UnresolvedCustC 
UnresolvedDevG 


Y 

JPredictedCustor 

optional 

Weight 

optional  numeric 

Freq 

optional  numeric 

By 

optional 

Personality:  Ordinal  Logistic 


Remove 


Construct  Model  Effects 


Add 

AvgAgeUnresolvedCustQuestionsAtC 

AvgWeeklyln  Person  Meetings 

AvgWeeklyTelecons 

ReqtsElicitationMethod 

UnresolvedCustQuestionsAtCoding 

UnresolvedDevQuestionsAtCoding 

Cross 

Nest 

Macros  v 

Degree  2 

Attributes  T 

Transform  2 

EH  No  Intercept 
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Parameter  Estimates 


Term 

Estimate 

Std  Error 

ChiSquare 

^Prob>ChiSq 

Intercept^] 

62  129936 

27.6369 

5.05/ 

0.0246s 

lntercept[2] 

110.845603 

40.444952 

7.51# 

0.0061s 

Intercept^] 

160.156478 

58.481434 

7  5(1 

0.0062s 

Intercept^] 

203.407054 

74.122921 

7  51 

0.0061s 

AvgAgeUnresolvedCustQuestionsAtCoding 

1.66957142 

0.8736435 

3.6* 

0.0560 

AvgWeeklylnPersonMeetings 

-31.445747 

12.098374 

6.7* 

0.0093s 

AvgWeeklyT  elecons 

-0.5290472 

0.7016369 

0.51 

0.4508 

RpqtsFlirjtatinnMpthnd[1  HH] 

27  4484984 

10  472716 

6  8j\ 

n  nnss* 

ReqtsElicitationMethod[2. 00] 

-3.9627567 

2.0628873 

3  691 

0.0547 

UnresolvedCustQuestionsAtCoding 

-1 .4306074 

0.5823547 

6.03  \ 

,  0.0140s 

UnresolvedDevQuestionsAtCoding 

-1.908446 

0.6884446 

7.68 

\  0.0056> 
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Model  Specification 


Select  Columns—i  —  Pick  Role  Variables 


PredictedCustorr 
^AvgAgeUnresolvt 
^AvgAgeUnresolvt 
^AvgWeeklylnPen 
iAvgWeeklvTelec 
■■■  ReqtsElicitationrv 
UnresolvedCustC 
^UnresolvedDevG 


Y 

JPredictedCustor 

optional 

Weight 

optional  numeric 

Freq 

optional  nu:  ^ 

By 

optional 

Personality:  Ordinal  Logistic 


Help 


Run  Model 


Construct  Model  Effects 


Add 

AvgAgeUnresolvedCustQuestionsAtCl 

Cross 

<2 

|AvgWeeklyTelecons  1; 

Nest 

□ 

UnresolvedCustQuestionsAtCoding 

Macros 

3 

UnresolvedDevQuestionsAtCoding 

Degree 

2 

Attributes 

© 

Transform 

© 

EH  No  Intercept 
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*  Model  Specification 


Select  Columns— i  — Pick  Role  Variables 


PredictedCuston 
^AvgAgeUnresolv* 
^AvgAgeUnresolvt 
^AvgWeeklylnPer; 
^AvgWeeklyTelec 
■■■  ReqtsElicitation^ 
^UnresolvedCustC 
^UnresolvedDevG 


Y 

JPredictedCustor 

optional 

Weight 

optional  numeric 

Freq 

optional  numeric 

By 

optional 

Personality:  Ordinal  Logistic 


Remove 


Construct  Model  Effects 


Add 

AvgAgeUnresolvedCustQuestionsAtC 

AvgWeeklylnPersonMeetings 

ReqtsElicitationMethod 

UnresolvedCustQuestionsAtCoding 

UnresolvedDevQuestionsAtCoding 

Cross 

Nest 

Macros  v  | 

Degree  2 

Attributes  ® 

Transform  s 

CH  No  Intercept 
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Parameter  Estimates 


Term 

Estimate 

Std  Error 

ChiSquare  Prob>ChiSq 

Intercept^] 

54.9797203 

24.345705 

5.10/  0.0239s 

Intercept^] 

100.794188 

33.940026 

8.8z 

1  0.0030* 

Intercept^] 

145.703314 

48.287961 

9.  i  d 

0.0025s 

Intercept^] 

186.096476 

61.562511 

9.1  J 

0.0025s 

AvgAgeUnresolvedCustQuestionsAtCoding 

1.56135331 

0.768001 

4.1 1 

0.0421s 

AvgWeeklylnPersonMeetings 

-28.947564 

10.104355 

8.21 

0.0042s 

ReqtsElicitationMethod[1 .00] 

25.2121153 

8.6077178 

8.5E 

\  0.0034* 

ReqtsElicitationMethod[2. 00] 

-3.5025523 

1.7280995 

4.11 

\  0.0427s 

UnresolvedCustQuestionsAtCoding 

-1.2619774 

0.4359331 

8.38  \  0.0038s 

UnresolvedDevQuestionsAtCoding 

-1.8034997 

0.6076278 

8.81 

\  0.0030/ 
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Whole  Model  Test 


Model 

Difference 

Full 

Reduced 


-LogLikelihood 

99.82037 

9.58574 
inn  4nmn 


:Square  (U) 

Observa 
Converged  by  Gradient 


DF  ChiSquare 

6  199.6407 
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^  JMP  -  [CustomerSatisfactionExerciseWithOrdinalLogTstTcRegressfon-  Fit  Ordina 


3  File  Edit  Tables  Rows  Cols  DOE  Analyze  Graph  Toots  View  Window  Help 


:  '£)  lifi  id  i 


Customers 


Windows 


-  ^  Custom erSatisfact 

j . 5"  Custom  erSatisf 

5"  Fit  Model 


!X  £- 


^Likelihood  Rato  Tests 
Wald  Tests 
Confidence  Intervals 
ROC  Curve 
Lift  Curve 


Profiler 

Save  1 

Script 

UDservations  for  ^ 

^  r 

► 

FredictedCustomerSat 


DF  Chi  Square  Proh>Chi 

6  199.6407  <00i 


0  9174 
90 


Converged  by  Gradient 


Lack  Of  Fit 


Source 

Lack  Of  Fit 

O'-k+i  i  r^i+  f-H  rJ 


DF  -LogLikelihood 

350  9.5857356 

occ  n  nnnnnnn 


ChiSquare 

19.17147 
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Parameter  Estimates 


Sensitivity  Indicator 


Desirability  Functions 

Maximize  Desirability 
Maximization  Options 
Maximize  for  each  Grid  Point 
Save  Desirabilities 
Set  Desirabilities 
Save  Desirability  Formula 


Reset  Factor  Grid 
Factor  Settings 


Output  Grid  Table 


Output  Random  Table 
Alter  Linear  Constraints 
Save  Linear  Constraints 
Default  N  Levels 


Interaction  Profiler 

ritfUTUiTurr  riumei 


Estimate  Std  Error  ChiSquare  Prob>ChiS 


54.9797203 

24.345705 

5.10 

0.023' 

100.794188 

33.940026 

8.82 

0.0031 

145.703314 

48.287961 

9.10 

0.0021 

186.096476 

61.562511 

9.14 

0.0021 

ding 

1.56135331 

0.768001 

4.13 

0.042- 

-28.947564 

10.104355 

8.21 

0.004; 

25.2121 153 

8.6077178 

8.58 

0.003* 

-3.5025523 

1.7280995 

4.11 

0.042; 

-1.2619774 

0.4359331 

8.38 

0.0031 

-1.8034997 

0.6076278 

8.81 

0.0031 

L-R 

Nparm  DF  ChiSquare 

Prob>ChiSq 

1 

1  6.09880179 

0.0135s 

1 

1  151.868665 

<.0001  * 

2 

2  111.379472 

<.0001  * 

1 

1  29.1823898 

<.0001  * 

1 

1  49.204595 

<.0001  * 

ttt 
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AvgAgeUnresolvedCustQ 

uestionsAtCoding 

AvgWeeklylnPers 

onMeetings 

ReqtsElicitati 

onMethod 

UnresolvedCustQues 

tionsAtCoding 

UnresolvedDevQues 

tionsAtCoding 

17.39 

0.11 

1.00 

22.63 

36.33 

17.39 

0.11 

1.00 

22.63 

40.585 

17.39 

0.11 

1.00 

22.63 

44.84 

17.39 

0.11 

1.00 

22.63 

49.095 

17.39 

0.11 

1.00 

22.63 

53.35 

17.39 

0.11 

1.00 

25.7125 

36.33 

17.39 

0.11 

1.00 

25.7125 

40.585 

17.39 

0.11 

1.00 

25.7125 

44.84 

17.39 

0.11 

1.00 

25.7125 

49.095 

17.39 

0.11 

1.00 

25.7125 

53.35 

17.39 

0.11 

1.00 

28.795 

36.33 

17.39 

0.11 

1.00 

28.795 

40.585 

17.39 

0.11 

1.00 

28.795 

44.84 

17.39 

0.11 

1.00 

28.795 

49.095 

17.39 

0.11 

1.00 

28.795 

53.35 

17.39 

0.11 

1.00 

31.8775 

36.33 

17.39 

0.11 

1.00 

31.8775 

40.585 

17.39 

0.11 

1.00 

31.8775 

k  44.84 

17.39 

0.11 

1.00 

31.8775 

^49.095 

17.39 

0.11 

1.00 

31.8775 

53.35 

Software  Engineering  Institute  CarnegieMellon 


Robert  Stoddard  and  Dave  Zubrow 

©2010  Carnegie  Mellon  University 


197 


SSTC  2010 


©  \ 

Probability(Predicted 

CustomerSat=1) 

Probability(Predicted 

CustomerSat=2) 

Probability(Predicted 

CustomerSat=3) 

ProbabilityfPredicted 

CustomerSat=4) 

Probability(Predicted 

CustomerSat=5) 

1 

0.99995809 

4. 191 42e-5 

0 

0 

0 

2 

0.91728006 

0.08271994 

0 

0 

0 

3 

0.00512779 

0.99487221 

0 

0 

0 

4 

2.39571  e-6 

0.9999976 

5. 329 1  e- 1 5 

0 

0 

5 

1.1 1354e-9 

1 

1 . 1 385e- 1 1 

0 

0 

6 

0.99795395 

0.00204605 

0 

0 

0 

7 

0.18480994 

0.81519006 

0 

0 

0 

8 

0.00010536 

0.99989464 

2.2204e-16 

0 

0 

9 

4.89789e-8 

0.99999995 

2.589e-13 

0 

0 

10 

2.2766&-1 1 

1 

5.5686e-10 

0 

0 

11 

0.90885654 

0.09114346 

0 

0 

0 

12 

0.00461353 

0.99538647 

0 

0 

0 

13 

2. 1 5433e-6 

0.99999785 

5.9952e-15 

0 

0 

14 

1 .001 35e-9 

1 

1.266e-t  1 

0 

0 

15 

4.6543e-1 3 

0.99999997 

2.72379e-8 

0 

0 

16 

0.16934269 

0.83065731 

0 

0 

0 

17 

0.00009475 

0.99990525 

2.2204e-16 

0 

0 

18 

4.4044e-8 

0.99999996 

2.8777e-13 

0 

0 

19 

2.0472e-1 1 

1 

6. 1 926e- 1 0 

0 

0 

20 

9.51 55e-1 5 

0.99999867 

1 .33229 e-6 

0 

0 

21 

0.00415062 

0.99584938 

0 

0 

0 
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Contact  Information 


SSTC2010 


Robert  W.  Stoddard 

Email:  rws@sei.cmu.edu 

Dave  Zubrow 

Email:  dz@sei.cmu.edu 


U.S.  mail: 

Software  Engineering  Institute 
Customer  Relations 
4500  Fifth  Avenue 
Pittsburgh,  PA  15213-2612 
USA 


World  Wide  Web:  Customer  Relations 

www.sei.cmu.edu  Email:  customer- 

www.sei.cmu.edu/contact.html  relations@sei.cmu.edu 

Telephone:  +1  412-268-5800 

SEI  Phone:  +1  412-268-5800 

SEI  Fax:  +1  412-268-6257 
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